Hyperdexterous surgical system

ABSTRACT

A hyperdexterous surgical system is provided. The system can include one or more surgical arms coupleable to a fixture and configured to support one or more surgical tools. The system can include an electronic control system configured to communicate electronically with the one or more robotic surgical tools. The control system can electronically control the operation of the one or more surgical tools. The system can include one or more portable handheld controllers actuatable by a surgeon to communicate one or more control signals to the one or more surgical tools via the electronic control system to operate the one or more surgical tools. The one or more portable handheld controllers can provide said one or more control signals from a plurality of locations of an operating arena, allowing a surgeon to be mobile during a surgical procedure and to remotely operate the one or more surgical tools from different locations of the operating arena.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

This application is a continuation application of U.S. Ser. No.14/510,465, which is a continuation of application of U.S. Ser. No.14/388,180 filed Sep. 25, 2014, which is a US National Phase ofInternational Application No. PCT/US2014/026115 filed Mar. 13, 2014designating the US and published in English on Sep. 25, 2014 as WO2014/151621, which claims the priority benefit under 35 U.S.C. § 119(e)of U.S. Provisional Application No. 61/791,248 filed Mar. 15, 2013, U.S.Provisional Application No. 61/906,802 filed Nov. 20, 2013, U.S.Provisional Application No. 61/908,888 filed Nov. 26, 2013, U.S.Provisional Application No. 61/915,403 filed Dec. 12, 2013, and U.S.Provisional Application No. 61/935,966 filed Feb. 5, 2014, all of whichare hereby incorporated by reference in their entirety and should beconsidered a part of this specification.

BACKGROUND Field

Surgical robots allow surgeons to operate on patients in a minimallyinvasive manner. The present application relates to surgical systems andmethods, and more particularly to a hyperdexterous surgical system withone or more hyperdexterous surgical arms and one or more hyperdexteroussurgical tools, and methods of operating the same.

Description of the Related Art

Currently, surgeons must select between discrete modes of minimallyinvasive surgery utilizing many techniques. Laparoscopic surgerygenerally falls in two categories: laparoscopic surgery with manualtools and laparoscopic surgery with robotic tools. In laparoscopicsurgery using manual tools, procedures are typically performed throughsmall incisions. The manual tools can be translated, rotated, and/ormoved about a fulcrum. For manual tools that rotate about a fulcrum, thesurgeon holds the handle of the tool. As the surgeon moves the handle inone direction, the distal end of the tool moves in another direction.The resulting motion of the distal end of the tool relative to themotion of the proximal end of the tool may not be natural, requiring thesurgeon to practice the technique.

The motions of the laparoscopic tool are captured by a laparoscopiccamera. The laparoscopic camera has a long shaft that is inserted intothe body through an incision just like a manual tool. The laparoscopiccamera is positioned to view the distal tips of the manual tools andcaptures the motion of the distal end of the tools. The displaytypically shows the motion of the tools relative to the frame ofreference of the camera. For manual tools that rotate about a fulcrum,the tool moves in a polar coordinate system which may not be readilyapparent based on the images of the laparoscopic camera.

Another mode of laparoscopic surgery is robotic surgery. In on-marketrobotic surgical systems, a large robotic arm controls a robotic tool.The tool is inserted into a small incision. The distal end of therobotic tool typically includes an end effector (e.g., a grasper,stapler, etc.) for performing a procedure within the body of thepatient. The end effector is translated in space, within the constraintsof the capabilities of the robotic arm. The surgeon typically controlsthe robotic arm from an immersive console that is remote from thepatient. The robotic tool is configured to do certain surgical taskswell, but is not well-suited for other surgical tasks.

In on-market surgical robotic systems, the motions of the robotic toolare generally captured by a robotic camera. The motions of the roboticcamera are controlled by a robotic arm, also under control of thesurgeon just like the robotic arms controlling the robotic tools. Thesurgeon can map the movements of his hand to the movement of the robotictool in the frame of reference of the camera. The motions of thesurgeon's hands are mapped to the distal end effectors of the robotictools within the frame of reference of the robotic camera. The frame ofreference is therefore limited to the view provided by the camera. Thedisplay typically shows the motion of the distal end of the robotictools relative to the frame of reference of the camera. The surgeon musttherefore create a mental model of the anatomy with the limitedinformation provided by the camera to control the robotic tools asdesired for a particular task. Due to his remote location, the surgeoncannot acquire additional views of the patient in order to augment hisunderstanding of the surgical space. This mode of operation is limitingfor large motions or motions where it is more natural to move withrespect to a frame of reference outside the body of the patient.Therefore, controlling the distal tips of the robotic tools relative tothe robotic camera frame of reference makes some aspects of the surgicalprocedure more natural as compared to laparoscopic surgery using manualtools. For example it may be easier to manipulate a needle holder toolin a suturing task. However, the limited frame of reference of therobotic camera makes some other aspects of the surgery less natural. Forexample, making large movements from one quadrant of the abdomen toanother, especially motions that involve the camera sweeping through anarc that includes the midline of the patient, are very challenging.These same motions can be accomplished in a natural manner with manualtools from a frame of reference external to the patient's body.

The on-market systems have complex mechanisms controlling the tool, forinstance controlling the rotation and translation of the tool. In somecurrent robotic systems, translation of the tool is achieved using acomplex and bulky series of nesting linear slides. In order to make thefull length of the tool shaft available for surgery, the slides areattached to the extreme proximal end of the tool. As a result, in anycondition except full extension of the tool into the body, thetranslation mechanism extends away from the patient's body. In thisposition, the translation mechanism is subject to interference withother components of the robotic arm or other robotic arms. The size ofthe rotation and translation mechanism does not allow close positioningof adjacent robotic arms, so in some cases, robotic tools are placedfurther apart. The translation mechanism imparts a high inertial load onthe robotic arm when the tool moves through pitch and yaw, therebynecessitating a larger, more powerful arm. The rotation and translationmechanisms add weight to the distal end of the robotic arm. The linkingsegments and the motors to control the linking segments must thereforebe larger in order to move the complex rotation and translationmechanisms controlling the robotic tool. Each additional segment andeach additional motor add weight that compounds the problem. The distalend of the robotic arm is heavy and has to be supported by increasinglymore powerful proximal joints to maintain adequate level of stiffness.

The robotic arms therefore are bulky and occupy the space surroundingthe patient. In cases where multiple robotic arms used to perform asurgical procedure, the arms must be carefully coordinated to avoidcollisions. Further, many additional steps are taken to reposition therobotic arm to avoid collisions between components of the robotic arm.Further, due to the angle of insertion, the size and design of therobotic arms and tools, and other factors, the robotic arm may be unableto reach certain locations, called dead zones. The large size of therobotic arm forces the surgical staff to plan the operation around therobotic arm. This leads to less flexibility and efficiency for surgicalprocedures. Additionally, on-market robotic arms are heavy. The designof the robotic systems requires specially designed operating arenas,already set up for the use of the robotic system. There is thus limitedflexibility in the setup of the operating room.

The surgeon is located remotely from the patient when using on-marketrobotic surgical systems, often sitting or standing at a remote console.Typically, the surgeon views the surgery site and tools through a viewerwhich provides an immersive experience. In some cases, communicationbetween the surgeon and the supporting staff is constrained or impededdue to the surgeon's position over the console. Teams that performrobotic surgery need to be highly trained and skilled since the surgeonis remote from the patient and unable to communicate with the staffdirectly. It takes months, in some cases years, of practice to achieve ahigh level of efficiency in situations where robotic surgery isperformed. This makes it difficult for members of the team to bereplaced. Additionally, from this remote location (at the console), thesurgeon cannot simultaneously use manual tools while controlling therobot arm.

Some tasks such as executing large scale motion of the robotic toolsfrom one surgical site to another surgical site in a patient's bodybecome more difficult due to the interference of components of therobotic arms. Some tasks easily performed with manual tools are morecomplex or impossible to perform with robotic tools. For example, insome cases, the robot simply does not have an end effector capable ofaccomplishing the task. Some tasks requiring tactile feedback, such aspalpation, cannot be done by the surgeon operating the robotic arm.Rather, the surgeon operating the robotic arm requires an assistant or asurgeon beside the operating table to assist in these types of tasks.

On-market robotic arms typically have two degrees of freedom. Typically,these two degrees of freedom come from a pitch mechanism and a rollmechanism. The robotic tool typically has four degrees of freedom. Therobotic tool can typically translate and rotate. The robotic tool cantypically pitch and yaw at the wrist. The on-market systems typicallythus have six degrees of freedom including the degrees of freedom fromthe robotic arm and the robotic tool.

The translation mechanism used by some robotic arms cannot rotate aboutthe shaft axis. To achieve rotation, these systems simply rotate justthe tool shaft independently of the translation mechanism. The cableswhich articulate the end effector twist during rotation, thus causingfriction and binding of the cables. This twisting causes a change oflength in the cables which must be compensated for by elasticity orslack in the system. This twisting also causes a limitation on the rangeof rotation, typically limited to approximately +/−270° of rotation.

One drawback of the current modes of minimally invasive surgerydiscussed above is that they are discrete. In order for the surgeon touse manual tools at the operating table, he or she cannot be controllingthe robotic arm at a remote console. In order for the surgeon to controlthe robotic arm at a remote console, he or she cannot be using manualtools at the operating table. The surgeon cannot simultaneously controlboth robotic tools and manual tools.

Another drawback of the current modes of minimally invasive surgery isthat they provide limited information to the surgeon. Typically thisinformation is limited to the view of a robotic camera. The surgeon isnot provided with information about additional constraints, such as thelocation of the patient, surgeon, or tools relative to the image fromthe camera. The surgeon is not provided with information to understandthe frame of reference of the camera without moving the tools and/ormoving the robotic camera. By moving the tools and viewing the image,the surgeon can create a mental model of the work space inside thepatient and the operating arena.

Another drawback with on-market robotic surgical systems is that they donot allow the surgeon the ability to reposition him or herself duringsurgery. The surgeon must remain at the immersive console to manipulatethe robotic tools to perform a surgical task with the end effectors ofthe robotic tools.

Another drawback of on-market robotic surgical systems is that they aretypically anchored to the ground and do not follow the orientation ofthe patient during the course of surgery. The position of the roboticarm and/or bed cannot be changed while the robotic arm is in use.Typically robotic arms are mounted to a horizontal level surface (e.g.,anchored to the floor) and the patient is placed on a horizontal levelsurface (e.g., bed). In some surgeries, it may be advantageous to angle(e.g., tilt) the body of the patient relative to the horizontal surface(e.g., lowering the head of the patient to have internal organs shifttoward the patient's head) based on the surgery to be performed.

Another drawback with on-market robotic arms is that accessing theworkspace may require the robotic arms to move through a very largerange of motion. The movement may be limited when multiple robotic armsare used for a single surgery. The chances of collision between therobotic arms or components of a single robotic arm increases. Thechallenge is to maximize the work space inside the body while maximizingthe free space outside of the patient, while also keeping the roboticsystem small and compact.

SUMMARY OF THE INVENTION

There is a need for a surgical system that overcomes the deficienciesdiscussed above with on-market robotic surgical systems and providesflexibility to surgeons when performing surgical procedures.

The hyperdexterous surgical system discussed below overcomes many of thedeficiencies discussed above and provides advantages over on-marketrobotic surgical systems. One advantage of the hyperdexterous surgicalsystem is that the hyperdexterous surgical system is small and compact,and therefore can be mounted in a variety of ways to a variety offixtures. One advantage of the hyperdexterous surgical system is thatthe hyperdexterous surgical arm can be mounted to follow an orientationof a patient during a surgical procedure, such as when the body of thepatient is tilted to facilitate conducting a particular surgicalprocedure (e.g., to shift internal organs in a way that provides betteraccess to the desired tissue or organ). One advantage of thehyperdexterous surgical system is the ability to use hyperdexteroussurgical tools and manual tools simultaneously by a surgeon whileoperating on a patient. Another advantage of the hyperdexterous surgicalsystem is that it is modular and thus provides flexibility in how thesurgical arena is set up prior to or during a procedure, and allows thefree space above the patient to be maximized. Still another advantage ofthe system is that it allows the surgeon to be mobile while performing asurgical procedure and to seamlessly move between using only manualtools, using manual and hyperdexterous surgical tools, and using onlyhyperdexterous surgical tools during the surgical procedure. Anotheradvantage of the system is that it provides the surgeon with additionalinformation that makes the operation of hyperdexterous surgical toolsmore natural. Still another advantage is that it provides the surgeonwith the ability to reposition him or herself during surgery to performa particular surgical task near the patient. For example, during thecourse of a surgical procedure, the surgeon may desire to manipulatetools from different positions based on the procedure to be done, or toreposition him or herself due to the manner in which a manual tool needsto be held. Still another advantage of the system is that the endeffector of a hyperdexterous surgical tool can reach disparate locationsinside the patient from a single entry point, such that the work spaceinside the patient's body is maximized. For example in abdominalsurgery, there may be a need to access all four quadrants of the abdomenfrom a single entry point. Further advantages of the hyperdexteroussurgical system will become apparent in the description provided herein.

In accordance with another aspect, the hyperdexterous surgical arm cancouple to a fixture (e.g., operating table, hospital bed, examinationtable, wall, floor, ceiling, table, cart, or dolly). The hyperdexteroussurgical arm can be supported by a support arm. The support arm can bemoved to position the hyperdexterous surgical arm. The support arm canbe moved to position the Remote Center. The hyperdexterous surgical armcan be supported by a horizontal position adjusting mechanism. Thehyperdexterous surgical arm can be supported by a vertical positionadjusting mechanism. The horizontal position adjusting mechanism and/orthe vertical position adjusting mechanism can be moved to position theRemote Center.

In accordance with another aspect, the hyperdexterous surgical systemcan enable the one or more hyperdexterous surgical arms to be angled(e.g., tilt) to follow an orientation of a patient during the course ofthe surgery. Typically the patient is placed on a horizontal levelsurface (e.g., bed). In some surgeries, it may be advantageous to angle(e.g. tilt) the body of the patient relative to the horizontal surface(e.g., lowering the head of the patient to shift internal organs towardthe head of the patient away from a surgical site for improved access tothe surgical site) based on the surgery to be performed. Thehyperdexterous surgical system thus enables the angling (e.g., tiltingfrom horizontal) of the hyperdexterous surgical arm during the procedurewith the hyperdexterous surgical arm in use.

In accordance with one aspect, the hyperdexterous surgical systemaccommodates the simultaneous use of a manual tool and a hyperdexteroussurgical tool by one operator, such as a surgeon. The simultaneous useof manual tools and hyperdexterous surgical tools can be in the sameworkspace inside the patient. The operator can control a manual toolwith one hand and a hyperdexterous surgical tool with the other hand.

The hyperdexterous surgical tool can include a tool shaft, a wrist andan end effector. The tool can have a motor pack at a proximal end orlocated at any point along the shaft of the tool. The motor pack caninclude a plurality of motors that actuate movement of a drive mechanismin the tool to effect motion of the end effector. In one embodiment, themotor pack can be removable.

In accordance with another aspect, the hyperdexterous surgical systemenables the operator to interact with the patient from multiplelocations, including at the patient's bedside, i.e., at the operatingtable, while operating the hyperdexterous surgical tools. Thehyperdexterous surgical system enables the operator to control one ormore hyperdexterous surgical tools, or simultaneously control ahyperdexterous surgical tool and a manual tool, at the patient's bedsidewhile positioned next to the patient.

In accordance with another aspect, the hyperdexterous surgical systemenables the operator to be mobile around the operating arena during theprocedure. The mobility allows the surgeon to find the optimal positionabout the patient to perform a surgical procedure and to reposition himor herself during the course of a surgery as needed or desired. Thehyperdexterous surgical system thus enables the operator to control ahyperdexterous surgical tool from a plurality of locations, includingfrom the patient's bedside and/or from a separate remote stand. Theoperator can relocate to a more optimal position to manipulate a manualtool and/or a hyperdexterous surgical tool.

In accordance with another aspect, the hyperdexterous surgical system ismodular, thereby enabling flexibility and versatility in arranging oneor more hyperdexterous surgical arms of the hyperdexterous surgicalsystem relative to the patient. Such flexibility provided by thehyperdexterous surgical system allows advantageous spacing of thehyperdexterous surgical arms. This flexibility also allows the operatingarena to be set up to conform to the patient or the environment prior tobeginning a surgical procedure, and to be modified during a surgicalprocedure, by adding or removing hyperdexterous surgical arms as needed.The flexibility provided by the modular aspect of the hyperdexteroussurgical system also enables more free space around the patient, whichlimits collisions between one or more hyperdexterous surgical arms. Saidfree space also allows the surgeon greater access to the patient, forexample to manipulate a manual tool from various positions (e.g.,simultaneously with a hyperdexterous surgical tool) or reposition him orherself relative to the patient during a surgery, such as when emergencyprocedures need to be performed on the patient. Said free space providedby the hyperdexterous surgical system also allows for positioning ofadditional hyperdexterous surgical arms, as well as allows for greaterrange of motion of the hyperdexterous surgical arms.

In accordance with another aspect, the size and/or weight of thehyperdexterous surgical arm is minimized. The size and/or weight of therotate/translate mechanism of a hyperdexterous surgical tool isminimized, which allows the size and/or weight of the hyperdexteroussurgical arm that supports the hyperdexterous surgical tool to beminimized. Minimizing the size and weight of the hyperdexterous surgicalarm allows the use of drive mechanisms, such as motors, that are lessbulky to effect movement of the hyperdexterous surgical arm. Further,the amount of power needed to power the drive mechanism of thehyperdexterous surgical arm is reduced. Additionally, the smaller sizeand/or weight of the hyperdexterous surgical arm allows for flexibilityin the mounting of the hyperdexterous surgical arm to a fixture. Due tothe smaller space taken up by hyperdexterous surgical system, theoperator can advantageously have more free space around the surgicalarena.

In accordance with another aspect, the hyperdexterous surgical systemfacilitates the surgeon's natural understanding of the motion of thehyperdexterous surgical tools, by augmenting the surgeon's understandingof the positioning of the tools. The hyperdexterous surgical systemprovides information regarding the positioning of the manual tools andhyperdexterous surgical tools within the workspace, inside the body of apatient. For example, the hyperdexterous surgical system can providevisual cues to the surgeon that help the surgeon understand theorientation and position of the hyperdexterous surgical tools relativeto the surgeon, allowing the surgeon to understand how thehyperdexterous surgical tools will move when actuated by the surgeon.The hyperdexterous surgical system enables the control of thehyperdexterous surgical tools to be adjusted based upon the preferencesof the operator. The hyperdexterous surgical system enables theinformation presented to the operator to be adjusted based upon thepreferences of the operator.

In accordance with another aspect, the hyperdexterous surgical systemreduces the dead zone, the region within the body inaccessible by thehyperdexterous surgical tool. The hyperdexterous surgical arm can bepositioned such that the dead zones can be placed away from thepatient's body. The hyperdexterous surgical system can be designed suchthat mounting the hyperdexterous surgical arm to minimize the dead zoneis easy to achieve. The hyperdexterous surgical system can be designedsuch that a neutral position and/or a zero position of thehyperdexterous surgical arm minimize the dead zone.

In accordance with another aspect, the hyperdexterous surgical arm canhave a redundant degree of freedom. The redundant degree of freedom canallow the hyperdexterous surgical arm to be placed in a variety ofdesired poses. The redundant degree of freedom can enable more freespace around the patient. The redundant degree of freedom can enable theplacement and use of more hyperdexterous surgical arms (e.g., aplurality of hyperdexterous surgical arms), within the space above thepatient. Additionally, the redundant degree of freedom can enable alarger workspace inside the patient. The redundant degree of freedom canlimit the number of self-collisions (between components of a singlehyperdexterous surgical arm) and other collisions (betweenhyperdexterous surgical arms, between hyperdexterous surgical arm andthe patient).

In accordance with one aspect, the hyperdexterous surgical arm can havethree degrees of freedom. The hyperdexterous surgical arm can have oneredundant degree of freedom compared with on-market systems. Thehyperdexterous surgical arm can have two roll axes. One of the two rollaxes can be a redundant roll axis. The hyperdexterous surgical arm canhave a redundant roll mechanism. The hyperdexterous surgical tool andthe rotate/translate mechanism can have four degrees of freedom. Thehyperdexterous surgical tool can rotate and translate. Additionally, thehyperdexterous surgical tool can pitch and roll (e.g., via a wrist). Thehyperdexterous surgical arm, hyperdexterous surgical tool androtate/translate mechanism can together provide seven degrees offreedom. In one embodiment, the hyperdexterous surgical arm,hyperdexterous surgical tool and rotate/translate mechanism can togetherprovide more than seven degrees of freedom. The hyperdexterous surgicalarm can have more than one redundant degree of freedom compared withon-market systems. The redundant degree of freedom can allow thehyperdexterous surgical arm to be placed in a variety of desired poses.Additionally, the redundant degree of freedom can allow thehyperdexterous surgical tool to be positioned in a desired orientationvia a variety of poses of the hyperdexterous surgical arm.

The hyperdexterous surgical arm can be positioned to establish a RemoteCenter. The Remote Center is the location where entry into the bodyoccurs. For the hyperdexterous surgical system, the Remote Center is alocation in space where the axes of rotation of the various roll andpitch mechanisms of the hyperdexterous surgical arm and the axis of thehyperdexterous surgical tool intersect. The Remote Center can be locatedat the incision of a patient. The shoulder roll mechanism can be placedbelow the Remote Center to position the dead zone outside the body ofthe patient.

In accordance with another aspect, the hyperdexterous surgical arm caninclude a pitch mechanism, a first roll mechanism, and a second rollmechanism. The axis of the first and second roll mechanism can passthrough the Remote Center. Additionally, an axis of a hyperdexteroussurgical tool coupled to the hyperdexterous surgical arm can passthrough the Remote Center. The hyperdexterous surgical arm can bearranged such that the vertical location of the second roll mechanismcan be at or below the Remote Center through which all the axes pass.The second roll mechanism is the redundant roll mechanism.

The hyperdexterous surgical arm can be arranged such that the secondroll mechanism can rotate at least up to +/−90° from an initialposition. In some embodiments, the hyperdexterous surgical arm can bearranged such that the second roll mechanism can rotate more than +/−90°from an initial position. The second roll mechanism can advantageouslyreach targets that are inaccessible or difficult to reach with anon-market robotic arm having only the pitch mechanism and one rollmechanism.

Due to the arrangement of the pitch mechanism, the first roll mechanism,and the second roll mechanism, the hyperdexterous surgical arm canassume various poses. The target location may be accessed by changingthe orientation of the pitch mechanism, the first roll mechanism, and/orthe second roll mechanism while maintaining Remote Center.

In accordance with another aspect, the hyperdexterous surgical systemincludes a rotate/translate mechanism that can impart rotation and/ortranslation on a hyperdexterous surgical tool. The hyperdexteroussurgical arm can be arranged such that the rotate/translate mechanism islocated proximate the Remote Center (e.g., within 3-5 inches, within 2-6inches, within 1-7 inches, less than 7 inches, less than 6 inches, lessthan 5 inches, less than 4 inches, less than 3 inches, less than 2inches, less than 1 inch). The rotate/translate mechanism can bearranged to have a limited contribution to rotational moment of inertiaof the hyperdexterous surgical arm. The rotate/translate mechanism canbe arranged to limit interference with adjacent hyperdexterous surgicalarms during movement. The rotate/translate mechanism can be arrangedsuch that it acts directly on the shaft of the hyperdexterous surgicaltool. The rotate/translate mechanism can be arranged such that itaccommodates different size shafts of the hyperdexterous surgical tools.The rotate/translate mechanism can have a smaller width than on-marketsystems, allowing hyperdexterous surgical tools of adjacenthyperdexterous surgical arms to be positioned close together.

The rotate/translate mechanism can be arranged such that the mechanicalenergy inputs for rotation and translation can be differential such thatrotation and/or translation are achieved by combined motion of the twomechanical energy inputs. The mechanical energy inputs for rotation andtranslation can be differential such that the power applied to themechanism is the combined power of the two input motors. Therotate/translate mechanism can be arranged such that it maintains abarrier between sterile components of the hyperdexterous surgical systemand non-sterile components of the hyperdexterous surgical system. Therotate/translate mechanism can be arranged such that the shaft positionof the hyperdexterous surgical tool is measured directly on the shaftthrough the resistance or capacitance of the shaft length outside thebody of the patient.

In accordance with another aspect, the hyperdexterous surgical systemcan include a control system. The hyperdexterous surgical arm can becontrolled by an input device. The hyperdexterous surgical tool can becontrolled by an input device. The position and orientation of the inputdevice can be tracked. The input device can be wireless or wired. Thehyperdexterous surgical system can include one or more input devices(e.g., two, three, four, five, six input devices, etc.).

The input devices can control one or more control points. The controlpoints are locations which have the capability to execute some motion.One or more control points can be located on the hyperdexterous surgicalarm. One or more control points can be located on the hyperdexteroussurgical tool. The conversion of movement of the input device tomovement of one or more control points may be independent of themovement of other control points. The conversion of movement of theinput device to movement of one or more control points may besynchronized with the movement of other control points. The operator cancontrol one or more control points simultaneously.

The controlled objects may be selected from the group comprising one ormore hyperdexterous surgical tools and/or one or more hyperdexteroussurgical arms. The control system of the hyperdexterous surgical systemcan convert the movement of the input device into movements of thecontrolled objects dependent on the zoom factor of images displayed onone or more displays.

The control system can include the application of constraints betweenthe one or more input devices and the one or more controlled objects.The control system can be arranged such that the constraints aremeasured quantities such as position or derived parameters such asdistance, velocity, force, and tension. The control system can bearranged such that the constraints can be different for each controlledobject. The control system can be arranged such that the constraints canbe the same for a group of controlled objects. Each hyperdexteroussurgical tool in the set can have an independent constraint. Theconstraint can be that one or more hyperdexterous surgical tools can bemanipulated together with a single input device.

The hyperdexterous surgical system can include an electronic controlsystem that communicates with the one or more hyperdexterous surgicalarms and/or one or more hyperdexterous surgical tools. Thehyperdexterous surgical system can include one or more input devicesthat communicate a signal with the control system. The signal from theinput devices can be transmitted within the operating arena. Forexample, the signal from the input devices can be transmitted from thebedside of a patient, allowing the operator to control hyperdexteroussurgical arm from the bedside of a patient. The control system cancommunicate a signal with the one or more hyperdexterous surgical armsand/or one or more hyperdexterous surgical tools from various locationswithin the operating arena.

In accordance with another aspect, the hyperdexterous surgical systemenables the control of a hyperdexterous surgical tool and a manual toolin frames of reference that are aligned, partially aligned orindependent of each other. The hyperdexterous surgical system enablescontrol of one or more hyperdexterous surgical tools in frames ofreference that are aligned, partially aligned or independent of eachother. The hyperdexterous surgical system provides information to theoperator regarding the frames of references.

In accordance with another aspect, the hyperdexterous surgical systemenables the movement of a hyperdexterous surgical tool to be locked tothe movement of a single tool. The hyperdexterous surgical systemenables the movement of one or more hyperdexterous surgical tools to belocked to the movement of a single hyperdexterous surgical tool. Thehyperdexterous surgical system enables the movement of one or morehyperdexterous surgical tools to be locked to the movement of a singlemanual tool.

In accordance with another aspect, the hyperdexterous surgical systemenables and disables motion of one or more hyperdexterous surgical toolswith a mechanism. The mechanism can be a clutch. The operation of themechanism can establish a frame of reference for the hyperdexteroussurgical tool. The operation of the mechanism enables the operator toestablish a new reference frame after the initial establishment of areference frame. The hyperdexterous surgical system can be arranged suchthat the frame of reference may be associated with the wrist or forearmof the operator's hand that is operating the input device. Thehyperdexterous surgical system can be arranged such that the frame ofreference may be associated with the wrist or forearm of the operator'shand that is operating the manual tool. The operator can manipulate oneor more hyperdexterous surgical tools in a frame of reference that isindependent of the frame of reference of the one or more manual tools.

In accordance with another aspect, the hyperdexterous surgical systemfacilitates an understanding of the frames of references. Thehyperdexterous surgical system can include a visualization system thataggregates information from one or more sources and provides one or moreimages to the surgeon. The information may be positional information ofthe surgeon, the patient, the hyperdexterous surgical arm, thehyperdexterous surgical tool, and/or the manual tool. The informationmay be positional information of control points. The image can bemanipulated to reflect the point of view of the surgeon. The image canbe updated to reflect the point of view of the surgeon as the surgeonmoves to another location.

The information presented by the visualization system may be live datafrom the cameras, data from pre-operative MRI, CT, ultrasound or otherimaging modality, and models of organs and other parts of the humanbody. The visualization system can be arranged such that the image canbe updated in real time as data from cameras is received. Thevisualization system can be arranged such that the image can be a blendof information from various sources. The blending and the types ofinformation to blend may depend on the zoom factor. The image can beupdated to display warnings related to the information (e.g.,non-real-time, not precisely aligned models, low-resolution data).

The visualization system can present an image on one or more displays.The image displayed on each display may be different. Each display maypresent different images (e.g., the location of the patient, thelocation of control points). The visualization system can be arrangedsuch that the images can be adjusted automatically dependent on the typeof manipulation being performed. The visualization system can bearranged such that the images may be rotated and oriented according tothe surgeon's location. The visualization system can be arranged suchthat during the zooming operation, the images can be blended totransition smoothly between a zoomed in image and a zoomed out image.The images may be controlled by the user input devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A schematically illustrates an interaction continuum.

FIG. 1B schematically illustrates scenarios along the interactioncontinuum.

FIG. 2 schematically illustrates one embodiment of a hyperdexteroussurgical system.

FIG. 3A-C schematically illustrates scenarios along the interactioncontinuum.

FIG. 4 schematically illustrates a hyperdexterous surgical arm coupledto a bed.

FIG. 5 schematically illustrates an embodiment of a support arm.

FIG. 6 schematically illustrates the multiple degrees of freedom of thehyperdexterous surgical arm of FIG. 4.

FIG. 7 schematically illustrates an embodiment of a hyperdexteroussurgical arm with rotation axes and a Remote Center.

FIG. 8 schematically illustrates the degree of freedoms of an embodimentof the hyperdexterous surgical arm of FIG. 7.

FIG. 9 schematically illustrates a zero position, an initial position ofa hyperdexterous surgical tool, and a target position.

FIG. 10 schematically illustrates the inability of a hyperdexteroussurgical tool to reach the target position due to the interferencebetween the pitch segment and the other portions of the hyperdexteroussurgical arm.

FIG. 11 schematically illustrates how the shoulder roll segment may beactivated to reach the same target point of FIG. 10.

FIG. 12 schematically illustrates the location of a dead zone for ahyperdexterous surgical arm.

FIG. 13A schematically illustrates an arrangement of a hyperdexteroussurgical arm to reach a target position.

FIG. 13B schematically illustrates another arrangement of ahyperdexterous surgical arm to reach the target position of FIG. 13A.

FIG. 14 schematically illustrates an embodiment of a hyperdexteroussurgical arm.

FIG. 15 schematically illustrates an embodiment of a hyperdexteroussurgical arm.

FIG. 16 schematically illustrates an embodiment of a hyperdexteroussurgical arm.

FIG. 17 schematically illustrates an embodiment of an asymmetricrotate/translate mechanism.

FIG. 18 schematically illustrates a top view of the asymmetricrotate/translate mechanism of FIG. 17.

FIG. 19 schematically illustrates a side view of the asymmetricrotate/translate mechanism of FIG. 17 with arrows showing translation ofa hyperdexterous surgical tool.

FIG. 20 schematically illustrates the top view of the asymmetricrotate/translate mechanism of FIG. 17 with arrows showing translation ofa hyperdexterous surgical tool.

FIG. 21 schematically illustrates the side view of the asymmetricrotate/translate mechanism of FIG. 17 with arrows showing rotation of ahyperdexterous surgical tool.

FIG. 22 schematically illustrates the top view of the asymmetricrotate/translate mechanism of FIG. 17 with arrows showing rotation of ahyperdexterous surgical tool.

FIG. 23 schematically illustrates an embodiment of a symmetricrotate/translate mechanism.

FIG. 24 schematically illustrates a side view of the symmetricrotate/translate mechanism of FIG. 23 showing the linear drive belts andthe rollers of FIG. 23.

FIG. 25 schematically illustrates a side view of the symmetricrotate/translate mechanism of FIG. 23 with arrows showing translation ofa hyperdexterous surgical tool.

FIG. 26 schematically illustrates the side view showing the linear drivebelts and the rollers of the symmetric rotate/translate mechanism ofFIG. 23 with arrows showing translation of a hyperdexterous surgicaltool.

FIG. 27 schematically illustrates the side view of the symmetricrotate/translate mechanism of FIG. 23 with arrows showing rotation of ahyperdexterous surgical tool.

FIG. 28 schematically illustrates an embodiment of a rotate/translatemechanism with a continuous belt drive mechanism.

FIG. 29 schematically illustrates an embodiment of a width adjustermechanism.

FIG. 30 schematically illustrates the side view of the width adjustermechanism of FIG. 29 in a new position.

FIG. 31 schematically illustrates the top view of the new position ofthe rollers of FIG. 30.

FIG. 32A is an embodiment of an input device.

FIG. 32B is an embodiment of an input device.

FIGS. 33A-33B schematically illustrate a virtual grip.

FIG. 34 schematically illustrates an embodiment of a control system of ahyperdexterous surgical system.

FIG. 35 schematically illustrates a block diagram of a control system.

FIG. 36 schematically illustrates an embodiment of a screenshot of adisplay.

FIG. 37 schematically illustrates a screenshot of a display.

FIGS. 38A-38B schematically illustrate a method of holding the tissue inconstant tension.

FIG. 39 schematically illustrates a method of using the hyperdexteroussurgical tools and a manual tool at the same time.

FIG. 40 schematically illustrates a screenshot of a display.

FIG. 41A schematically illustrates two frames of reference that areoriented the same way.

FIG. 41B schematically illustrates the motion as perceived by theobservers placed at the locations specified in FIG. 41A.

FIG. 41C schematically illustrates two frames of reference that are notoriented the same way.

FIG. 41D schematically illustrates the motion as perceived by theobservers placed at the locations specified in FIG. 41C.

FIG. 42A schematically illustrates an operator controlling ahyperdexterous surgical tool and a manual tool.

FIG. 42B schematically illustrates a screen shot of a display.

FIG. 42C schematically illustrates an operator controllinghyperdexterous surgical tools.

FIG. 43 schematically illustrates a screen shot of a display.

FIGS. 44A-44C schematically illustrate different images presented on adisplay.

FIG. 45 schematically illustrates a screen shot of a display.

DETAILED DESCRIPTION

The term “hyperdexterous” is a combination of the ordinary meaning of“hyper” and “dexterous”; hyper meaning over or above, and dexterousmeaning skillful or adroit in the use of the hands or body. Ahyperdexterous surgical system as used herein enables interactionsbetween a surgeon and the patient along an interaction continuum, asfurther described below, and provides increased versatility with respectto the surgical procedures that can be performed. The hyperdexteroussurgical system enhances the ability of the surgeon to interact with apatient and includes several features which combine to produce a morenatural, more interactive, and more versatile surgical system. Theversatility of the hyperdexterous surgical system is illustrated byvarious aspects of the system, such as for example its modularity, whichallows the use of one or more hyperdexterous surgical arms and to movethe hyperdexterous surgical arm out of the way to utilize only manualsurgical tools, its enabling of the surgeon to be mobile in the surgicalarena during a surgical procedure, and its enabling of the surgeon tosimultaneously operate a hyperdexterous surgical tool and a manual toolwhile being able to maneuver between multiple bedside locations of thepatient to an optimal position for a particular surgical task. With thisfeature, and with other features to be described below, a hyperdexteroussurgical system has more versatility than on-market purely “robotic”surgical systems.

The nature of the hyperdexterous surgical system is illustrated, amongother ways, by providing the surgeon with a variety of information(e.g., via displays) that allow the surgeon to readily understand thepositioning of the hyperdexterous surgical tools relative to the patientso as to naturally understand how the tools will move when actuated. Theinteractive nature of the hyperdexterous surgical system is illustrated,among other things, by the ability of the surgeon to selectively controla plurality of hyperdexterous surgical tools with user input devices, aswell as the ability to move between different frames of reference duringa surgical procedure (e.g., between an immersive frame of referenceinside the patient's body and a frame of reference outside the patient'sbody), allowing the surgeon to reposition himself or herself during aprocedure, all the while remaining aware of the position and orientationof the tools relative to the surgeon.

Introduction

The hyperdexterous surgical system described herein provides afundamentally different conceptual framework from existing on-marketrobotic surgical systems in that, among other things, it enables asurgeon to simultaneously use manual and hyperdexterous surgical toolswhile at the patient's bedside. This ability to be at the patient'sbedside, i.e., beside the operating table, provides several advantages:from improved communication with the surgical team; to direct monitoringof the patient; to facilitating tool exchanges (e.g., between manual andhyperdexterous surgical tools). Moreover, the hyperdexterous surgicalsystem, as discussed below, includes several subsystems that togetherprovide a flexible, more natural, and more interactive system thatadvantageously allows surgeons to perform surgical procedures seamlesslyalong an interaction continuum between using only hyperdexteroussurgical tools, simultaneously using a combination of manual andhyperdexterous surgical tools, and using only manual tools as desired bythe surgeon or required by the surgical task.

One advantageous aspect of the hyperdexterous surgical system is thesize of the hyperdexterous surgical arm, which is smaller and morecompact than those of on-market systems, and which allows increasedflexibility in how the arm is mounted relative to the patient—whether ona cart, or dolly, or wall or ceiling of the operating room, or directlyto the patient's bed. The smaller size of the hyperdexterous surgicalarm also allows for the hyperdexterous surgical system to be modular,where the number of hyperdexterous surgical arms used can vary asdesired by the surgeon depending on the surgical need. The small size ofthe hyperdexterous surgical arms additionally provide for increased freespace above the patient, which facilitates the surgeon's ability to workfrom a bedside location. Indeed, as noted above, one inventive aspect ofthe hyperdexterous surgical system is that it allows the surgeon tooperate along an interaction continuum, and in one scenario the surgeoncan move the hyperdexterous surgical arms out of the way and use onlymanual tools. This ability to maximize the free space (e.g., move thehyperdexterous surgical arms out of the way, small size of the arm), isenhanced by advantageously providing a hyperdexterous surgical arm withthree degrees of freedom including a redundant roll, as discussed inmore detail below. Additionally, the redundant roll of thehyperdexterous surgical arm is advantageously positioned so as to ensurethat a dead zone for the hyperdexterous surgical tool is located outsidethe body of the patient, thereby allowing for increased access of thehyperdexterous surgical tool within the workspace in the body.

Further, the rotation and translation mechanism for the hyperdexteroussurgical tool advantageously facilitates the small size of thehyperdexterous surgical arm, as discussed below. Indeed, therotate/translate system is smaller in diameter, lighter and more compactthan mechanisms that impart rotation and translation for on-marketsystems, which allows the hyperdexterous surgical arm that supports therotate/translate mechanism to be smaller.

Another advantageous and interrelated aspect of the hyperdexteroussurgical system is the ability it provides to the surgeon to move aroundfreely and position him or herself in an optimal position near thepatient. This ability is provided by the hyperdexterous surgical systemin several ways, including allowing the surgeon to control thehyperdexterous surgical arm with one or more handheld portable inputdevices that communicate the movements of the surgeon's hands to acontrol system that controls the operation of the hyperdexteroussurgical arm and hyperdexterous surgical tool. In one embodiment, thehandheld portable input devices are wireless. The hyperdexteroussurgical system advantageously provides for tracking of the handheldportable input devices, as well as features (e.g., a clutch) to preventunintended motion of the hyperdexterous surgical arms or hyperdexteroussurgical tools due to movements of the surgeon.

Still another advantageous and interrelated part of the hyperdexteroussurgical system is the control system, which communicates the surgeon'scommands (e.g., via the user input devices) to the hyperdexteroussurgical tools and hyperdexterous surgical arms and facilitates how thesurgeon interacts with the hyperdexterous surgical tools andhyperdexterous surgical arms. Another advantageous and interrelated partof the system is the visualization system, which aids the surgeon inmanipulating the hyperdexterous surgical tools within the patient's bodyfrom various frames of reference (e.g., immersive, bird's eye viewoutside the patient's body). The control system and visualization systemmay work together to enhance the surgeon's ability in performing asurgical procedure by providing a variety of information (e.g., visualcues) that allows the surgeon to naturally recognize the positioning ofthe hyperdexterous surgical tools and manual tools.

Each of the components or subsystems of the hyperdexterous surgicalsystem have advantages over corresponding components in on-marketsystems. Additionally, taken together the components and subsystemsprovide a hyperdexterous surgical system that provides a completelydifferent paradigm for surgical procedures that enhances the ability ofthe surgeon to interact with a patient in a more natural, moreinteractive, and more flexible manner. The hyperdexterous surgicalsystem will now be described in more detail.

FIG. 1A shows the interaction continuum of the hyperdexterous surgicalsystem. The right end of the continuum illustrates physical interactionsbetween the body of the surgeon and the body of the patient. The farright end of the continuum includes the surgeon moving tissue by hand.The use of a manual tool such as a scalpel is less physicallyinteractive than moving tissue by hand. The use of a laparoscopic toolsuch as a scalpel is less physically interactive than moving tissue byhand. The left end of the continuum illustrates the use of ahyperdexterous surgical arm to control end effectors. Along thecontinuum, the operator can use manual tools and/or hyperdexterous toolsin various combinations. The hyperdexterous surgical system enables anoperator 1, such as a surgeon, to work anywhere along the continuum.

At the right end of the continuum, the surgeon is in close proximity tothe patient in order to physically manipulate the tissue. The surgeon isable to directly touch and feel tissue at the surgical site. At the leftend of the continuum, the surgeon interacts with the patient remotely bymanipulating user input devices that serve as proxies for the realtools. The surgeon manipulates end-effectors such as graspers bymanipulating user input devices. Many scenarios occur along theinteraction continuum.

Embodiments herein describe the use of a hyperdexterous surgical systemthat can be used by an operator. The operator may be a surgeon, amedical assistant, staff, medical examiners, or any other personoperating the hyperdexterous surgical system. An operator is not limitedto a medical professional qualified to practice surgery, but includesany operator trained to operate the hyperdexterous surgical system.

FIG. 1B replicates the interactive continuum of FIG. 1A in more detail.FIG. 1B shows various methods of using a hyperdexterous surgical system,such as the hyperdexterous surgical system 100 shown in FIG. 2. On theright side of FIG. 1B, an operator 1 may perform a surgical step withmanual tools 350. In this scenario, a hyperdexterous surgical arm 200 ofthe hyperdexterous surgical system 100 may be moved away from the workspace. The operator 1 may refer to a display 702 which provides imagesof the surgery. On the left side of FIG. 1B, the operator 1 interactswith an input device 500 to control the hyperdexterous surgical arm 200.The operator 1 may refer to a display 600 which provides images of thesurgery.

In the middle of this continuum, different scenarios may occur. Onescenario is illustrated as Scenario 1 in FIG. 1B. In this scenario, theoperator 1 may simultaneously use one or more hyperdexterous surgicaltools 300 and one or more manual tools 350 (e.g., at the same time, inthe same workspace). The surgeon can manipulate one or more of thehyperdexterous surgical tools 300 with one or more of the input devices500. The operator 1 may refer to a display 702 which provides images ofthe surgery. Another scenario is illustrated as Scenario 2 in FIG. 1B.In this scenario, the operator 1 may manipulate the hyperdexteroussurgical arm 200 by hand. This scenario may occur for example whenexecuting large scale motions of the hyperdexterous surgical arms 200.Though FIG. 1B shows only one hyperdexterous surgical arm 200, thehyperdexterous surgical system 100 can have a plurality ofhyperdexterous surgical arms 200, as shown in FIG. 2.

In another scenarios, not shown, the operator 1 can insert the manualtool 350 into a trocar 302 (shown in FIG. 2) supported by thehyperdexterous surgical arm 200. The trocar 302, with the manual tool350 inserted therein, may be manipulated by hand. The third scenario maybe useful when it may be difficult to maneuver the manual tool 350 onlywith the hands. Some examples where such situations may be encounteredis when the patient is obese or if the angle of entry into the patientis awkward. In such situations the hyperdexterous surgical arm 200holding the trocar 302 may act as a power assist to position the manualtool 350. In other words, the hyperdexterous surgical arm 200 holdingthe trocar 302 may counter the forces that are applied on the manualtool 350 by the patient's body, and can enhance maneuverability of themanual tool 350. The versatility of the hyperdexterous surgical system100 advantageously allows all such combinations.

FIGS. 3A-3C shows scenarios wherein the operator 1 selects the type oftools to be used in a surgical procedure. FIG. 3B is similar to Scenario1 and FIG. 3C is similar to the left side of the continuum shown in FIG.1B. The figures show three dimensional depiction of use thehyperdexterous surgical system 100 during a surgical procedure on apatient 2.

In FIG. 3A, the hyperdexterous surgical system 100 includes twohyperdexterous surgical arms 200, each coupled to a hyperdexteroussurgical tool 300. The operator 1 controls a hyperdexterous surgical arm200 with an input device 500 held in his right hand. The operator 1controls another hyperdexterous surgical arm 200 with an input device500 held in his left hand. The input devices 500 move the hyperdexteroussurgical arms 200 and/or the hyperdexterous surgical tools 300 inresponse to the operator's 1 movement. The hyperdexterous surgicalsystem 100 includes a display 600. The display 600 may allow theoperator 1 to establish constraints to be applied to the hyperdexteroussurgical system 100. For example, the display 600 may allow the operator1 to establish associations (e.g., a pairing) between the input devices500 and the controlled objects (e.g., the hyperdexterous surgical arms200 and/or the hyperdexterous surgical tools 300). The display 600 mayprovide images of the surgery.

In FIG. 3B, the hyperdexterous surgical system 100 includes ahyperdexterous surgical arm 200 coupled to a hyperdexterous surgicaltool 300. The operator 1 controls the hyperdexterous surgical arm 200with an input device 500 held in his right hand. The input device 500moves the hyperdexterous surgical arm 200 and/or the hyperdexteroussurgical tool 300 in response to the operator's 1 movement. The operator1 controls a manual tool 350 with his left hand. As illustrated in FIG.3B and discussed herein, the hyperdexterous surgical system 100advantageously enables the operator 1 (e.g., surgeon) to simultaneouslycontrol a hyperdexterous surgical tool 300 and a manual tool 350.

In FIG. 3C, the hyperdexterous surgical system 100 includes twohyperdexterous surgical arms 200, each coupled to a hyperdexteroussurgical tool 300. The operator 1 can control the hyperdexteroussurgical arms 200 with one or more input devices 500. The input devices500 can be the input devices shown in FIG. 3A. The input devices 500 canbe controllers as shown in FIG. 3C. The one or more input devices 500can be mounted or otherwise fixed near the display 600. In anotherembodiment, the one or more input devices 500 can be handheld portableinput devices, such as those shown in FIG. 3A, and the operator 1 cansupport his or her arms on a support bar or rest bar of the stand whilestanding or sitting at the stand and operating the handheld portableinput devices. Like the input devices 500 shown in FIG. 3A, the one ormore input devices 500 shown in FIG. 3C moves the hyperdexteroussurgical arms 200 and/or the hyperdexterous surgical tools 300 inresponse to the operator's 1 movement.

System Overview

The hyperdexterous surgical system 100 can include many components thatcan work together to achieve benefits described herein, such asenhancing the ability of the surgeon to interact with the patient byproviding a more natural, more interactive, and more versatile surgicalsystem. The hyperdexterous surgical system 100 can include one or morehyperdexterous surgical arms 200, and each hyperdexterous surgical arms200 can manipulate a hyperdexterous surgical tool 300. Thehyperdexterous surgical system 100 includes a control system anddisplays 600, 702 which provide the operator with visual cues that aidthe surgeon in controlling the operation of the one or morehyperdexterous surgical tools 300 and the manual tools 350. Theoperator, such as a surgeon can optionally manipulate the hyperdexteroussurgical tool 300 and the manual tools 350 simultaneously at variouslocations in the operating arena.

FIG. 2 shows one embodiment of a hyperdexterous surgical system 100. Thehyperdexterous surgical system 100 includes one or more hyperdexteroussurgical arms 200. Each hyperdexterous surgical arm 200 can support ahyperdexterous surgical tool 300 (e.g., via a trocar 302). Thehyperdexterous surgical system 100 can include one or more manual tools350. The manual tool 350 can be used simultaneously with thehyperdexterous surgical tool 300 by the operator (e.g., surgeon). Thehyperdexterous surgical tool 300 can be controlled by an input device500. The input device 500 can take many forms including a pincher 502(see FIG. 32A) and a controller 514.

The hyperdexterous surgical system 100 can include a control system totranslate movements from the input devices 500 to movements of thehyperdexterous surgical arms 200 and hyperdexterous surgical tool 300.The operator 1 can select which input device 500 controls whichhyperdexterous surgical tool 300 or hyperdexterous surgical arms 200.The control system 400 can include a computer 402, one or more cables406 and/or a power supply 404.

The hyperdexterous surgical arm 200 can be coupled with a fixture (e.g.,operating table, hospital bed, examination table, wall, floor, ceiling,table, cart, dolly). In one embodiment, where the fixture is a cart ordolly, the fixture can be anchored (e.g., temporarily) to the floor. Thehyperdexterous surgical system 100 may include mechanisms that couple orhold the hyperdexterous surgical arm 200 to the fixture. In theembodiment of FIG. 2, the fixture is a bed or operating table 102.

The hyperdexterous surgical arm 200 and the hyperdexterous surgical tool300 can be controlled by a control system 400, a schematic of which isshown in FIGS. 34-35. The algorithms that guide the control systemand/or any computations performed by the control system may be stored bythe computer 402. The control system 400 can translate user commands tomotion of the hyperdexterous surgical arm 200 and/or motions of thehyperdexterous surgical tool 300. The computer 402 may be connected tothe power supply 404. The computer 402 may be connected to a clutch 112which may take the form of a foot pedal. The computer 402 may includecables 406 that connect the computer 402 to other components.

The hyperdexterous surgical system 100 can include one or more inputdevices 500. The input device 500 can communicate with the controlsystem 400, either through a wired or wireless connection. As used inembodiments herein, the term “wireless” encompasses all forms ofwireless communication, including, but not limited to, infrared (IR),radiofrequency (RF), microwave, and ultrasonic. The input device 500 cansend control signals to the appropriate motors within the hyperdexteroussurgical arm 200 and the hyperdexterous surgical tools 300 via thecontrol system 400 (e.g., by communicating a signal from a transmitterin the input device 500 to a receiver of the control system 400). Theinput devices 500 can be handheld and/or portable devices thatadvantageously allow the operator 1, such as a surgeon, to move aboutthe bedside of the patient 2 during a procedure. The input devices 500can allow the operator 1 to control the hyperdexterous surgical arm 200and/or the hyperdexterous surgical tools 300 from one or more locations(e.g., a plurality of locations). Some of the locations may be at thebedside of a patient 2. The hyperdexterous surgical system 100 caninclude the clutch 112. The clutch 112 can be used to engage ordisengage one or more hyperdexterous surgical tools 300. The clutch 112can be a foot pedal, as shown in FIG. 2.

With continued reference to FIG. 2, the hyperdexterous surgical system100 can include a user interface sub-system 605. The user interfacesub-system 605 can include an input device (e.g., controller 514). Theuser interface sub-system 605 can include a platform 602 which caninclude features such as a horizontal resting bar 603. The userinterface sub-system 605 can include a display 600. The display 600 caninclude a touch screen 604. The display 600 can be interactive andreceive an input from the operator 1. The display 600 can be used tocontrol the control system 400. The display 600 can be located remotelyfrom the patient. In some embodiments, the display 600 is mounted ontothe platform 602, as shown in FIG. 2. In some embodiments, the display600 can be affixed to the body of the operator 1, such as the surgeon.

In some embodiments, the user interface sub-system 605 can include aninput device 500 (e.g., a wired controller). The input device 500 may bea controller 514 mounted to the platform 602. The user interfacesub-system 605 allows an operator 1, such as a surgeon, to control theinput device 500 in close proximity to the display 600, as shown in FIG.2.

The display 600 allows the operator 1 to perform many functionsincluding pairing a hyperdexterous surgical tool 300 with an inputdevice 500 so that the operator 1 can operate the paired hyperdexteroussurgical tool 300 with the input device 500. The display 600 can allowthe operator 1 to control one or more hyperdexterous surgical tools 300with the one or more input devices 500. The display 600 also allows theoperator 1, such as a surgeon, to pair a hyperdexterous surgical arm 200with an input device 500 so that the operator 1 can operate the pairedhyperdexterous surgical arm 200 with the input device 500. The display600 can allow the operator 1 to control one or more hyperdexteroussurgical arms 200. The user interface 600 can show or illustrate a mapof the one or more input devices 500 and the one or more controlledobjects, such as the one or more hyperdexterous surgical arms 200 orhyperdexterous surgical tools 300.

With continued reference to FIG. 2, a visualization system 700 caninclude one or more displays 702. The display 702 can displayinformation about the one or more hyperdexterous surgical arms 200, theone or more hyperdexterous surgical tools 300, the patient, or any otherinformation that may be relevant to the surgeon or surgical team. Thedisplay 702 can show images as seen by a camera 304 (shown schematicallyin FIG. 35) or other visualization devices, such as images of thehyperdexterous surgical tools 300 that are held by the hyperdexteroussurgical arms 200, or images of a manual tool 350 held by the operator(e.g., surgeon). The camera 304 can be controlled by the control system400. The camera 304 can be considered a hyperdexterous surgical tool 300and moved by a hyperdexterous robotic arm 200. In one embodiment, thecamera 304 can be controlled by the input device 500 via the controlsystem 400, which enables the operator 1, such as the surgeon, toposition the camera 304 as needed. The hyperdexterous surgical system100 can include multiple displays 702 positioned at various locationsthroughout the operating arena. Additionally, the displays 600, 702 canshow the same information or different information.

As shown in FIG. 2, the hyperdexterous surgical system 100 can be usedwith one or more manual tools 350 (e.g., a plurality of manual tools350). One manual tool 350 is shown in FIG. 2. The manual tool 350 can beutilized in the same work space as the one or more hyperdexteroussurgical tools 300. One or more manual tools 350 can be used because thehyperdexterous surgical system 100 advantageously allows the operator 1to stand right by the patient 2, as discussed previously. One or moremanual tools 350 can be used because the hyperdexterous surgical arm 200is compact, thereby freeing up the space around the patient 2. Theredundant roll mechanism and the placement of the redundant rollmechanism, described herein, also maximizes the free space around thepatient 2. Therefore, the operator 1 can simultaneously manipulatehyperdexterous surgical tools 300 and manual tools 350 without collidinginto other components of the hyperdexterous surgical system 100. Theoperator 1, not shown in FIG. 2, may stand by the bedside, and have theability to choose to control one or more hyperdexterous surgical tools300 (e.g., via the input devices 500), one or more manual tools 350, orany combination of hyperdexterous surgical tools 300 and manual tools350.

As shown in FIG. 2, the hyperdexterous surgical arm 200 can be coupledto a hyperdexterous surgical tool 300. Accordingly, the system 100 canhave one or more (e.g. a plurality) of hyperdexterous surgical arms 200and one or more (e.g. a plurality) of hyperdexterous surgical tools 300.In some embodiments, the hyperdexterous surgical tool 300 is insertedinto a trocar 302. The trocar 302 can be coupled to the hyperdexteroussurgical arm 200 (e.g., affixed, integrally formed with, held by, etc.).The hyperdexterous surgical arm 200 can support and manipulate thehyperdexterous surgical tools 300 through the trocar 302. In someembodiments, the one or more hyperdexterous surgical arms 200 can insertone or more hyperdexterous surgical tools 300 through an incision in apatient 2, as shown in FIGS. 3A-C. The hyperdexterous surgical arm 200and the hyperdexterous surgical tool 300 can have one or more motors(e.g., electrical motors) at various locations, as discussed furtherbelow. The motors facilitate the placement of the hyperdexteroussurgical tools 300 appropriately in the operating work space, inside thepatient 2. The hyperdexterous surgical arm 200 and the hyperdexteroussurgical tool 300 can be powered by the power supply 404. In someembodiments, the one or more hyperdexterous surgical tools 300 can bedisposable. In some embodiments, at least a portion of thehyperdexterous surgical tools 300 can be capable of being sterilized(e.g., reusable).

Mounting the Hyperdexterous Surgical Arm

The hyperdexterous surgical system 100 can provide a mounting to supportthe hyperdexterous surgical arm 200. The mounting enables thepositioning of the hyperdexterous surgical arm 200 and/or thehyperdexterous surgical tools 300 relative to the patient. Thehyperdexterous surgical arm 200 can be mounted to a number of fixtures,which may be movable or fixed. The flexibility in mounting thehyperdexterous surgical arm 200 and/or the hyperdexterous surgical tools300 provides versatility in designing the operating arena and the freespace outside the patient.

FIGS. 2 and 4 show embodiments of the hyperdexterous surgical arm 200mounted to a fixture. The various components allow the hyperdexteroussurgical arm 200 to be positioned relative to the patient 2. The variouscomponents also allow the hyperdexterous surgical arm 200 to avoidcollisions with other hyperdexterous surgical arms 200. Thehyperdexterous surgical arm 200 can be positioned to facilitate accessto the patient 2. The hyperdexterous surgical arm 200 can be positionedto permit the operator 1 to use hyperdexterous surgical tools 300 andmanual tools 350 simultaneously. The flexibility of the hyperdexteroussurgical system 100 including the support arm 106, the elevator 120, andthe carriage 130 allows the positioning of a Remote Center 250 (see FIG.7) for the hyperdexterous surgical arm 200. Once the Remote Center 250is established, the hyperdexterous surgical arm 200 can be manipulatedwhile maintaining the Remote Center 250.

As shown in FIG. 2, the hyperdexterous surgical system 100 can include aplurality of mounting poles 104. The hyperdexterous surgical system caninclude any number of mounting poles 104 (e.g., one, three, four, etc.),but two are shown in FIG. 2 for illustrative purposes. Each mountingpole 104 can support a hyperdexterous surgical arm 200. FIG. 2 showseach mounting pole 104 only holding one hyperdexterous surgical arm 200,but each mounting pole 104 can optionally support any number ofhyperdexterous surgical arms 200 (e.g., one, two, three, four, etc.).The hyperdexterous surgical system 100 is advantageously modular innature. This modularity allows the users, such as a surgical team, toconfigure the hyperdexterous surgical system 100 most efficiently forthe type of procedure being performed. Such modularity also allows theteam to add or remove mounting poles 104 and/or hyperdexterous surgicalarms 200 during a surgery. The modularity permits the hyperdexteroussurgical system 100 to be configured in various ways.

The mounting pole 104 may be supported by a movable fixture (e.g., adolly, a hand-truck or a small cart). The mounting pole 104, and allassociated hyperdexterous surgical arms 200 and support arms 106, may bemounted to the moveable fixture at a location remote from the operatingarena. The movable fixture may be transported into the operating arenabefore or during the surgery. If additional hyperdexterous surgical arms200 are needed during surgery, the hyperdexterous surgical arm 200 canbe mounted quickly and easily onto the fixture. In some embodiments, ifadditional hyperdexterous surgical arms 200 are needed during surgery,additional hyperdexterous surgical arms 200 mounted on movable fixturesmay be transported into the operating arena. The moveable fixture and/orthe mounting pole can be anchored to another fixture (e.g., the floor)to enhance stability. The mounting pole 104 may be supported by animmobile fixture (e.g., bed, floor, wall, or ceiling).

The mounting pole 104 can be attached to a clamp 108, as shown in FIG.2. The clamp 108 can be connected to the fixture. In the illustratedembodiment, the fixture is a bed 102, though as discussed above, othersuitable fixtures can be used. In some embodiments, the clamp 108 can becoupled to one or more rails 110 of the bed 102. Other attachingmechanisms are possible. The mounting pole 104 may be placedsubstantially vertically (e.g., at ninety degrees) relative to thefixture (e.g., bed 102). The mounting pole 104 may be placed at otherangles, such as 15 degrees, 30 degrees, 45 degrees, 60 degrees, relativeto the fixture (e.g., bed 102). The mounting pole 104 can be placed atother angles based on the orientation of the patient.

The hyperdexterous surgical arm 200 can be directly or indirectlyattached to the fixture (e.g., bed 102). In one embodiment, the mountingpole and/or the support arm 106 is excluded and the hyperdexteroussurgical arm 200 can be coupled directly to the mounting pole 104 or toa portion of the fixture (e.g., bed 102). In some embodiments, thehyperdexterous surgical system 100 can be detached from the fixture(e.g., bed 102 and/or rail 110).

Referring to FIG. 4, the mounting pole 104 can be coupled to componentsthat permit horizontal movement (e.g., parallel to the bed 102) orvertical movement (e.g., perpendicular to the bed 102). The elevator 120allows the placement of the support arm 106 and/or the hyperdexteroussurgical arm 200 along the length of the mounting pole 104. One or moreelevators 120 may be coupled to the mounting pole 104, as shown. Eachelevator 120 may be connected to an additional support arm 106 and/or anadditional hyperdexterous surgical arm 200. Accordingly, the mountingpole 104 can optionally support multiple support arms 106, each supportarm 106 coupled to a hyperdexterous surgical arm 200. The elevators 120may provide alternative vertical locations at which the hyperdexteroussurgical arm 200 may be coupled to the mounting pole 104.

With continued reference to the embodiment illustrated in FIG. 4, themounting pole 104 can be coupled to a carriage 130. The carriage 130 maybe coupled to an adaptor 132. The carriage 130 and the adaptor 132 mayform a slide assembly that allows the carriage 130 to slide linearlyalong the adaptor 132. The adaptor 132 can be coupled to the fixture(e.g., bed 102 and/or the rail 110). The carriage 130 may be directlycoupled to the fixture (e.g., bed 102 and/or rail 110) without theadaptor 132. The carriage 130 and the adaptor 132 permit the movement ofthe mounting pole 104, the support arm 106, and the hyperdexteroussurgical arm 200 in the generally horizontal direction. In someembodiments, the mounting pole 104 directly couples to the fixture(e.g., bed 102 and/or the rails 110) without the carriage 130 and/or theadaptor 132. The mounting pole 104 can be arranged such that themounting pole 104 slides linearly along the fixture (e.g., bed 102and/or the rails 110).

Referring to FIG. 4, the mounting pole 104 can be coupled to componentsthat permit movement in other directions. For example, the mounting polecan be coupled to a slide (not shown) that moves orthogonal to thehorizontal and vertical direction (e.g., extends outward from thefixture). The slide can be a drawer mounted to the fixture. For example,in embodiments where the fixture is the bed 102 the hyperdexteroussurgical arm 200 can be moved laterally away from a side of the bed 102(e.g., via a slidable drawer).

The hyperdexterous surgical system 100 can include mechanisms thatcouple or hold the hyperdexterous surgical arms 200 upright. FIGS. 2 and4 shows the hyperdexterous surgical arm 200 optionally coupled to asupport arm 106. The support arm 106 can be a passive arm, lackingmotors or other electrical features. As shown in FIG. 5, the support arm106 has a first end 114 and a second end 116. The first end 114 caninclude a bracket 118, such as a u-shaped bracket, that can be coupledto the hyperdexterous surgical arm 200. Other connections known in theart can also be utilized. The bracket 118 can couple to thehyperdexterous surgical arm 200 at the base of the hyperdexteroussurgical arm 200, for example near a shoulder roll mechanism 202 (seeFIG. 7), described further below. The second end 116 can be coupled tothe elevator 120. The second end 116 can rotate about a center ofrotation 122.

The support arm 106 may include one or more centers of rotation thatallow the support arm 106 to rotate. The support arm 106 shown in FIG. 5has three centers of rotation 122. The centers of rotation 122 rotateabout an axis in the direction of the arrows, as shown in FIG. 5. Thesupport arm 106 may include one or more tilt axes 124 which allow aportion of the support arm 106 to tilt. As shown in FIG. 5, the supportarm 106 has one tilt axis 124 that allows a hyperdexterous surgical arm200 coupled to the support arm 106 to tilt. The centers of rotation 122and/or the tilt axis 124 allow the one or more links 126 of the supportarm 106 and the bracket 118 to be rotated and positioned. The centers ofrotation 122 may rotate the links 126 of the support arm 106 in the sameplane, or in different planes, or in some combination of planes.

The support arm 106 can be passive. The operator 1 can move the supportarm 106 by hand to position the support arm 106. The operator 1 can movethe support arm 106 by hand to establish the Remote Center 250,described herein. In some embodiments, the support arm 106 can beactive. In such an embodiment, the support arm 106 can include one ormore motors to move joints of the support arm 106. The operator 1 canmove the support arm 106 via the motors to establish the Remote Center250.

The hyperdexterous surgical system 100 provides flexibility inpositioning the hyperdexterous surgical arm 200 and/or thehyperdexterous surgical tool 300. The flexibility is advantageouslyenhanced by the centers of rotation 122 and/or tilt axes 124 of thesupport arm 106, shown in FIG. 5. The flexibility can be enhanced by theability to move (e.g., vertically) the elevator 120 along the mountingpole 104. The flexibility can be enhanced by the ability to move (e.g.,horizontally) the carriage 130 along the adaptor 132.

Referring now to FIG. 6, the support arm 106, the elevator 120, and thecarriage 130 can facilitate the positioning of the hyperdexteroussurgical arm 200. Arrow 134, Arrow 136, Arrow 138 and Arrow 140demonstrate the centers of rotations and tilt axes of the support arm106. Arrow 142 demonstrates the generally vertical direction theelevator 120 may be positioned along the mounting pole 104. Arrow 144demonstrates the generally horizontal direction the carriage 130 maymove along the adaptor 132 in relation to the fixture (e.g., bed 102and/or rails 110). In one embodiment, the mounting pole 104 may bevertical, and the adaptor 132 may be horizontal; hence Arrow 142 may bevertical and Arrow 144 may be horizontal. All of the arrows or degreesof freedom can be configured differently than shown in FIG. 6 (e.g. thesupport arm 106 can have more or fewer centers of rotations or tiltaxes).

The hyperdexterous surgical system 100 can enable the one or morehyperdexterous surgical arms 200 to be angled (e.g., tilt) to follow anorientation of a patient during the course of the surgery. Typically thepatient is placed on a horizontal level surface (e.g., bed). In somesurgeries, it may be advantageous to angle (e.g. tilt) the body of thepatient relative to the horizontal surface (e.g., lowering the head ofthe patient to shift internal organs toward the head of the patient awayfrom a surgical site for improved access to the surgical site) based onthe surgery to be performed. The hyperdexterous surgical system 100enables the angling (e.g., tilting from horizontal) of thehyperdexterous surgical arm 200 so that it follows the orientation ofthe patient 2 (e.g., the hyperdexterous surgical arm 200 is mounted tofollow the orientation of the patient 2). The support arm 106, theelevator 120, the carriage 130 and the slide (not shown) can facilitatethe tilting of the hyperdexterous surgical arm 200.

The position of the hyperdexterous surgical arm 200 in the work spacemay be tracked. In some embodiments, the position is tracked by couplingabsolute encoders (not shown) at each joint of the hyperdexteroussurgical arm 200. In some embodiments, a position sensor (such as anoptical tracker) is mounted at the base of the hyperdexterous surgicalarm 200. The position sensor can provide the position of thehyperdexterous surgical arm 200 relative to a ground reference point(not shown). The position sensor and/or the encoders can be utilized totrack the position of the hyperdexterous surgical arm 200. Further, theposition sensor and/or the encoders can be utilized to track theposition of the hyperdexterous surgical tool 300. One skilled in the artmay utilize others suitable sensors, mechanism or methods of trackingcomponents of the hyperdexterous surgical system 100. The hyperdexteroussurgical system 100 can have a global tracker that tracks thehyperdexterous surgical arm 200, hyperdexterous surgical tool 300, andadditional components of the hyperdexterous surgical system 100 (e.g.,the operator 1, the input devices 500).

The Hyperdexterous Surgical Arm

The hyperdexterous surgical arm 200 used with the hyperdexteroussurgical system 100 can have a redundant degree of freedom. Theredundant degree of freedom can advantageously allow the hyperdexteroussurgical arm 200 to be placed in a variety of desired poses.Additionally, the redundant degree of freedom can advantageously enablemore free space around the patient. The redundant degree of freedom canenable the use of more hyperdexterous surgical arms 200 (e.g., aplurality of hyperdexterous surgical arms 200). The redundant degree offreedom can enable the placement of more hyperdexterous surgical arms200 within the free space above the patient 2. The redundant degree offreedom can also enable a larger workspace inside the patient. Theredundant degree of freedom can reduce self-collisions (betweencomponents of a single hyperdexterous surgical arm 200) and othercollisions (between hyperdexterous surgical arms 200, betweenhyperdexterous surgical arm 200 and the patient 2).

Redundancy is defined as follows: “When a manipulator can reach aspecified position with more than one configuration of the linkages, themanipulator is said to be redundant.” P. J. KcKerrow, Introduction toRobotics (Addison-Wesley Publishing Co, Sydney, 1991).

FIG. 7 shows one embodiment of a hyperdexterous surgical arm 200. Thehyperdexterous surgical arm 200 can be used with the hyperdexteroussurgical system 100 described herein. The hyperdexterous surgical arm200 can have three degrees of freedom.

The redundant degree of freedom, in relation to on-market surgicalsystems, can be provided by the shoulder roll mechanism 202. Theshoulder roll mechanism 202 can be located near a bottom of thehyperdexterous surgical arm 200. The redundant degree of freedomprovides additional flexibility and advantages. An advantage provided bythe redundant degree of freedom is that the hyperdexterous surgical arm200 can access a larger work space and can access additional anatomicaltargets that on-market robotic arms with two degrees of freedom cannotreach. The redundant degree of freedom allows the hyperdexteroussurgical arm 200 to maintain a tip position of the hyperdexteroussurgical tool 300 and Remote Center 250 while reconfiguring thecomponents of the hyperdexterous surgical arm 200 external to the body.These different poses enable the hyperdexterous surgical arm 200 toavoid collisions with the patient 2, other tools or other objects in thesurgical arena.

In some embodiments, the redundant degree of freedom, in relation toon-market surgical systems, is provided by a redundant pitch mechanism(not shown). The redundant pitch mechanism can be located anywhere onthe hyperdexterous surgical arm 200. The redundant pitch mechanism canbe located near a bottom of the hyperdexterous surgical arm 200. Theredundant pitch mechanism can have a pitch axis that intersects theRemote Center, as described herein. The redundant degree of freedomprovided by a redundant pitch mechanism can have the same advantages ofthe redundant degree of freedom provided by the redundant roll mechanism202 described herein. The hyperdexterous surgical tool 300 and therotate/translate mechanism 208 can have four degrees of freedom (e.g.,rotate, translate, pitch, yaw). The rotate/translate mechanism 208 canrotate and translate the tool 300. The hyperdexterous surgical tool 300can pitch and roll. Therefore, the hyperdexterous surgical arm 200 andthe hyperdexterous surgical tool 300 can have seven degrees of freedomin total. The hyperdexterous surgical arm 200 and the hyperdexteroussurgical tool 300 can have more than seven degrees of freedom (e.g.,eight degrees of freedom, nine degrees of freedom, etc.). Thehyperdexterous surgical arm 200 and the hyperdexterous surgical tool canhave additional degrees of freedom (e.g., provided by end effectors,such as graspers, flexible elbows). Typically, on-market surgicalrobotic systems have a robotic arm with two degrees of freedom (pitchand roll) and the tool has four degrees of freedom (rotate, translate,pitch, yaw), such that on-market robotic surgical systems typically havea total of six degrees of freedom.

With continued reference to FIG. 7, the shoulder roll mechanism 202provides one degree of freedom. The main roll mechanism 204 provides onedegree of freedom. The pitch mechanism 206 provides one degree offreedom. The rotate/translate mechanism 208 and hyperdexterous surgicaltool 300 provide four degrees of freedom (rotate, translate, pitch,yaw). There are four mechanisms which contribute to the dexterity of thehyperdexterous surgical arm: (1) the shoulder roll mechanism 202; (2)the main roll mechanism 204; (3) the pitch mechanism 206; and (4) therotate translate mechanism 208. Incorporating a second roll mechanism,the shoulder roll mechanism 202, provides a redundant degree of freedom(e.g. a seventh degree of freedom) as compared to on-market surgicalsystem.

The shoulder roll mechanism 202 has shoulder roll axis 244. The mainroll mechanism 204 has main roll axis 240. The pitch mechanism 206 haspitch axis 228. The axes 228, 240, 244 of the mechanisms 202, 204, 206intersect at a common point. In FIG. 7, this point is labeled the RemoteCenter 250.

Referring to FIG. 8, the pitch mechanism 206 allows the hyperdexteroussurgical arm 200 to rotate the hyperdexterous surgical tool 300 aboutthe pitch axis 228. The pitch axis 228 passes through the Remote Center250. Arrow 230 illustrates the arc representing the path of thehyperdexterous surgical tool 300 about the pitch axis 228. The pitchmechanism 206 has three centers of rotation, 232, 234, and 236 Thecenters of rotation 232, 234, and 236 are mechanically linked so as tocreate motion about the pitch axis 228.

Referring still to FIG. 8, the pitch mechanism 206 has three segments,the pitch segment 224, the pitch segment 226, and the pitch segment 227.The pitch segments 224, 226, and 227 of the pitch mechanism 206 can havemany configurations, such as a 2-bar, 3-bar, or 4-bar linkage, or cablelinkages. In some embodiments, bands or belts constrain the relativeangles between the pitch segments 224, 226, and 227. The pitch segment224 and the pitch segment 226 may collapse on top of each other or havea small angle between each other while rotating the hyperdexteroussurgical tool 300 around the pitch axis 228. When the pitch segments areclose to the collapsed position, the main roll mechanism 204 andproximal end of the trocar 302 can be brought close together.Alternatively, the pitch segment 224 and pitch segment 226 can alsoextend out, or have a large angle between each other so that thedistance between the main roll mechanism 204 and the proximal end of thetrocar 302 are spaced further apart.

The rotations and motions of the roll mechanisms 202, 204 are shown inFIG. 8. The arrow 238 shows the rotation of the main roll mechanism 204about main roll axis 240. The arrow 242 shows the rotation of theshoulder roll mechanism 202 about the shoulder roll axis 244. In someembodiments, the shoulder roll mechanism 202 can rotate at least up to+/−90° from an initial position. In other embodiments, the shoulder rollmechanism 202 can rotate more than +/−90° from an initial position. Themain roll axis 240 and the shoulder roll axis 244 intersect at theRemote Center 250 as shown in FIG. 8. The main roll segment 222 of themain roll mechanism 204 and/or shoulder roll segment 220 of the shoulderroll mechanism 202 can have any size, shape and/or number of segments.As shown, a shoulder roll segment 220 couples the shoulder rollmechanism 202 to the main roll mechanism 204 and a main roll segment 222couples the main roll mechanism 204 to the pitch segment 224.

One embodiment of the arrangement of the various segments of thehyperdexterous surgical arm 200 is shown in FIG. 7. One end of the firstsegment 218 can optionally be coupled to a fixture (e.g., the bed 102),the support arm 106 or other support objects within the operating arena,as discussed above. The other end of the first segment 218 is coupledwith the shoulder roll mechanism 202. The shoulder roll mechanism 202 isconnected to the main roll mechanism 204 with one or more segments 220.One or more segments 222, 224 connect the main roll mechanism 204 withthe pitch mechanism 206. One or more segments 226, 227 can connect thepitch mechanism 206 with the trocar 302.

In typical minimally invasive surgery, a small incision is made on thepatient's body through which the tools are passed into the body. Forexample, in abdominal surgery an incision is placed on the abdominalwall. To reduce the risk of harm to the patient, it is desirable tominimize movements that involve translation along the surface of thebody at the point of entry into the body as these types of movements maycause tearing of the tissues at the point of entry. Thus in minimallyinvasive procedures, it is desirable for the tool shaft to always passthrough a constant point. The Remote Center may be located at the pointof entry of the tools into the body. The hyperdexterous surgical tools300 can be pivoted about this point by the hyperdexterous surgical arm200 without tearing the tissue at the point of entry. The mounting ofthe hyperdexterous surgical arm 200 relative to the patient 2 canestablish the Remote Center 250 at the incision.

The arrangement of the axes allows the mechanisms 202, 204, 208 toachieve the desired position of the hyperdexterous surgical tool 300while the location of the Remote Center 250 is held constant. The RemoteCenter 250 may correspond with the location of an incision on a patientor the location of the entry point of a hyperdexterous surgical tool 300into the body as noted above. The Remote Center 250 can advantageouslybe held constant in order to reduce the risk of harm or injury to apatient. During surgery, for example during abdominal surgery, theRemote Center 250 may be placed at the abdominal wall. This location canbe a gateway for tools to enter the abdominal cavity. As differentpositions and orientations of a hyperdexterous surgical tool 300 aredesired, the shoulder roll mechanism 202, the main roll mechanism 204,and the pitch mechanism 206 may be activated in such a manner that thehyperdexterous surgical tool 300 pivots in the allowable degrees offreedom about the Remote Center 250.

The location of the Remote Center 250 may be constrained by the anatomyof the patient. The flexibility of the hyperdexterous surgical system100 advantageously allows the efficient placement of the one or morehyperdexterous surgical arms 200 and/or the hyperdexterous surgical tool300 in relation to the patient, the one or more operators 1 such assurgeons, the one or more assistants, and/or other objects or componentsfound within the operating arena.

FIG. 9 illustrates a hyperdexterous surgical arm 200 in an initialposition, referred to herein a zero position 246. An example of atargeted position, the target tool tip position 248, is shown. In thisexample, the target tool tip position 248 is directly in front of themain roll mechanism 204 and is collinear with the main roll axis 240.One way to try to reach the target tip position 248 is to collapse pitchsegment 224 and pitch segment 226 as shown in FIG. 10. However evenafter collapsing the pitch segments 224, 226, the hyperdexteroussurgical tool 300 is unable to reach the target tool tip position 248(see FIG. 10). This may occur, for example, because the end of the rangeof the pitch motion is encountered.

The shoulder roll mechanism 202 provides the redundant degree of freedomenabling the hyperdexterous surgical arm 200 to reach the target tooltip position 248, as shown in FIG. 11. The shoulder roll mechanism 202is a redundant roll mechanism. The shoulder roll mechanism 202 providesthe redundant degree of freedom as compared with on-market roboticsystems. Viewing the hyperdexterous surgical arm 200 from the targettool tip position 248, the shoulder roll segment 220 is partly rotatedclockwise and the main roll segment 222 is partly rotatedcounterclockwise, from the zero position 246, shown in FIG. 9. Byrotating the shoulder roll mechanism 202 and the main roll mechanism204, the target tool tip position 248 is now accessible. Thus it may beseen that a second roll mechanism, such as the shoulder roll mechanism202 shown in FIGS. 9-11, with an axis of rotation that intersects theaxis of rotation of another roll mechanism increases the dexterity ofthe hyperdexterous surgical arm 200. The shoulder roll mechanism 202 hasan axis of rotation 244 that intersects the axis of rotation 240 of themain roll mechanism 204, as shown in FIG. 8, therefore increasing themaneuverability and/or dexterity of the hyperdexterous surgical arm 200.

Singularities, Dead Zones, Free Space, Backlash

The design of the hyperdexterous surgical arm 200 provides significantattributes to the hyperdexterous surgical system 100. The location ofthe Rotation Center 250 relative to the hyperdexterous surgical arm 200permits dead zones to be placed away from the patient. The small size ofthe hyperdexterous surgical arm 200 enables the maximizing of free spacearound the patient, which facilitates the simultaneous use of manualtools and/or hyperdexterous surgical tools. The redundant degree offreedom provided by the shoulder roll mechanism 202 can increase theperformance of the system in such areas as lowering the effect ofbacklash and improving the practical bandwidth of the hyperdexteroussurgical arm 200.

The hyperdexterous surgical arm 200 may advantageously avoidsingularities during operation. A singularity is defined as thecollinear alignment of two or more axes. This condition may result inunpredictable motion and velocities of the hyperdexterous surgical arm200. When two axes align, rotation about either axis is not unique. Inother words, motion along one degree of freedom is lost.

Referring back to FIG. 10, the pitch segment 224 and the pitch segment226 are collapsed relative to each other. In such positions, the toolshaft axis 252 and the main roll axis 240 subtend an acute angle. If thetool shaft axis 252 and the main roll axis 240 were aligned, then therotation about the main roll axis 240 would be identical to the rotationabout the tool shaft axis 252. In such positions, movement about eitheraxis imparts the same motion to the hyperdexterous surgical tool 300. Insuch positions, control of the end effector 306 is not optimal becauseone degree of freedom is lost. FIG. 11 shows that the shoulder rollmechanism 202 can rotate the hyperdexterous surgical arm 200 such thatthe tool shaft axis 252 and the main roll axis 240 are not aligned. Theadditional degree of freedom offered by the shoulder roll mechanism 202prevents the hyperdexterous surgical system 100 from losing one degreeof freedom when axes align, substantially align, or are in nearalignment. In this way, the hyperdexterous surgical arm 200 may bedesigned to avoid singularities during the operation of thehyperdexterous surgical arm 200.

The hyperdexterous surgical arm 200 may advantageously minimize a deadzone. A dead zone is defined as one or more regions inaccessible by thehyperdexterous surgical tool 300. The dead zone 254 can be created byinterference between components of the hyperdexterous surgical arm 200.For example, as shown in FIG. 12, the dead zone 254 can be createdbecause of interference between the proximal end of the hyperdexteroussurgical tool 300 and the shoulder roll mechanism 202. The dead zone 254shown in FIG. 12 may be eliminated by making the pitch segments 224, 226and/or 227 with a different size, shape or number of segments (e.g.,longer to fit around the proximal end of the hyperdexterous surgicaltool 300). However, this may increase the size and/or weight of thehyperdexterous surgical arm 200. The dead zone 254 shown in FIG. 12 maybe eliminated by making the proximal end of hyperdexterous surgical tool300 a different size or shape (e.g., shorter to fit inside the shoulderroll mechanism 202).

The hyperdexterous surgical arm 200 may be designed so that one or moredead zones 254 occur outside the body of the patient 2, such that thedead zone does not limit the functionality of the hyperdexteroussurgical arm 200. The hyperdexterous surgical arm 200 is therefore ableto position the hyperdexterous surgical tool 300 anywhere within theworkspace, inside the patient's body.

Referring to FIG. 12, the main roll mechanism 204 is rotated such thatthe hyperdexterous surgical tool 300 now faces upwards. The shoulderroll mechanism 202 interferes or otherwise limits the movement of theproximal end of the hyperdexterous surgical tool 300 such as to create adead zone 254. The cross hatched area illustrating the dead zone 254shows positions the distal tip of the hyperdexterous surgical tool 300cannot achieve due to the obstruction of the proximal end of thehyperdexterous surgical tool 300 with the shoulder roll mechanism 202.FIG. 12 shows the body of the patient 2. As shown, the dead zone 254 isplaced outside and upwards away from the body of the patient 2. Theinability of the distal tip of the hyperdexterous surgical tool 300 toreach points within the dead zone 254 shown in FIG. 12 will not impactthe use of the hyperdexterous surgical tool 300 in surgical procedures.The dead zone 254 shown in FIG. 12 includes positions where surgery isnot performed.

As discussed above, the hyperdexterous surgical arm 200 can be mounted(e.g., via support arm 106, mounting poles 104, etc.) so that the deadzone occurs outside of the body. In some embodiments, the shoulder rollmechanism 202 may be located below the Remote Center 250 as shown inFIG. 7. In some embodiments, the shoulder roll mechanism 202 is closerto the fixture (e.g., hospital bed) to which the hyperdexterous surgicalarm 200 is mounted as shown in FIG. 12. By orienting the shoulder rollmechanism 202 as low as possible relative to the Remote Center 250, thedead zones 254 are advantageously placed up and away from the body ofthe patient 2. Referring back to FIG. 6, the shoulder roll mechanism 202is closer to the bed 102 or horizontal surface upon which the patient 2is placed. The ability to position the hyperdexterous surgical arm 200by positioning the support arm 106, the mounting pole 104, the elevator120, the carriage 130 and/or the adaptor 132 advantageously providesadditional flexibility in the placement of the dead zone 254. In someembodiments, the shoulder roll mechanism 202 may be located above theRemote Center 250 (e.g., due to mounting, positioning of patient ontheir side).

In some embodiments, the hyperdexterous surgical arm 200 is small insize. The hyperdexterous surgical arm 200 can in some embodiments weighless than 10 pounds, less than 8 pounds, less than 6 pounds, less than 4pounds, less than 3 pounds. The hyperdexterous surgical arm 200 can beless than 24 inches long, less than less than 22 inches long, less than20 inches long, less than 16 inches long, less than 14 inches long, lessthan 12 inches long. In one embodiment, the hyperdexterous surgical arm200 can be compact when in a collapsed configuration.

The small size of the hyperdexterous surgical arm 200 enables more freespace around the patient 2. The free space enables the surgeon tomanipulate a manual tool 350 from various positions. The free spaceenables the surgeon to reposition himself at multiple locations duringsurgery. The free space enables the operator to use manual tools 350concurrently with use of the hyperdexterous surgical arm 200. The freespace permits easier physical access to the patient when necessary.

Advantageously, the operator 1 is able to access the patient to use amanual tool 350 while simultaneously using the hyperdexterous surgicalarm 200 to control the hyperdexterous surgical tool 300. The operator 1may prefer to have free space to manipulate the required tools or tomove to a more optimal position with respect to the patient 2. Thehyperdexterous surgical arm 200 can be moved to a different positionwhile maintaining the Remote Center 250. The different position mayallow greater access to the patient. The shoulder roll mechanism 202provides the ability to move the hyperdexterous surgical arm 200 todifferent positions while maintaining the Remote Center

FIG. 13A shows an initial position of hyperdexterous surgical arm 200.The figure shows the target tool tip position 248 of the hyperdexteroussurgical tool 300. In FIG. 13B, the location and orientation of thehyperdexterous surgical tool 300 remains the same. The hyperdexteroussurgical arm 200 has assumed a different position. Viewing thehyperdexterous surgical arm 200 from the target tool tip position 248,the shoulder roll segment 220 is partly rotated counterclockwise and themain roll segment 222 is partly rotated clockwise from the positionshown in FIG. 13A. Thus along with the rotate/translate mechanism 206which imparts translation and rotation along the toll shaft axis 252,the hyperdexterous surgical tool 300 may be able to maintained at thesame target tool tip position 248 with various poses of thehyperdexterous surgical arm 200. The operator 1 can place thehyperdexterous surgical arm 200 in an optimal position for a procedure,as needed.

The shoulder roll mechanism 202 provides the ability to move thehyperdexterous surgical arm 200 to different positions to avoid otherrestrictions. Restrictions may be imposed in a surgical environment by avariety of factors. These factors include the body habitus of thepatient 2, limitations of objects in the operating arena based onphysical dimensions and movability, and the presence of otherinstruments near or in the work space. The shoulder roll mechanism 202may help to work around these restrictions. The hyperdexterous surgicalarm 200 may be positioned to minimize or eliminate the effects of therestrictions.

The redundant degree of freedom provided by the shoulder roll mechanism202 (as compared with on-market systems) will increase the performanceof the hyperdexterous surgical system 100 in such areas as lowering theeffect of backlash and improving or increasing the practical bandwidthof the system. Bandwidth is the ability of a hyperdexterous surgicaltool 300 to faithfully follow the motion of an input device 500. Higherbandwidths may allow the hyperdexterous surgical tool 300 to acceleratefaster. For instance, in a low bandwidth system, if an operator 1 movesthe input device 500 quickly or at a rapid speed, the hyperdexteroussurgical tool 300 may not be able to follow the motion of the inputdevice 500. Backlash is the amount of “play” in a mechanical system.

In systems that have a fixed Remote Center 250, bandwidth and backlashare generally inversely related. For example, if the end effector 306 ofthe hyperdexterous surgical tool 300 is very close to the Remote Center250, the main roll mechanism 204 and/or the shoulder roll mechanism 202moves a large amount to cause a small movement of the end effector 306.This movement stresses the bandwidth limitations of the hyperdexteroussurgical arm 200, but is favorable from an actuator backlashperspective. As another example, if the shaft of the hyperdexteroussurgical tool 300 is extended far into the body of the patient 2 andaway from any singularities, more demand is placed on the main rollmechanism 204 and/or the shoulder roll mechanism 202 to effect a changeof the end effector 306. The bandwidth is appropriate but the backlashmay become significant. The parameters such as bandwidth and backlashmay vary considerably from region to region within the work space

Sweet spots are regions where the hyperdexterous surgical system 100 iscontrolled and parameters such as bandwidth and backlash are withinacceptable parameters. Factors including bandwidth, backlash, mechanicallimitations imposed by the design of the system, location ofsingularities (where two or more degrees of freedom coincide) may impactthe sweet spot of the system. Designers attempt to have the sweet spotdefine a region as large as possible for the task to be performed.

In some embodiments, the bandwidth and backlash are optimized for allregions within the work space. The bandwidth and backlash are notoptimal around singularities. For example, a singularity may exist whenthe tool shaft axis 252 aligns with the main roll axis 240, as shown inFIG. 10. The end effector 306 may be more difficult to control. Theshoulder roll mechanism 202 may prevent singularities by providing anadditional degree of freedom. The shoulder roll mechanism 202 mayprovide additional poses of the hyperdexterous surgical arm 200 andprovide additional ways to arrange segments of the hyperdexteroussurgical arm 200. The shoulder roll mechanism 202 can provide a greaterworking area over which the bandwidth and backlash are within acceptableparameters.

Alternative Arms

The hyperdexterous surgical arm can have many configurations. FIG. 1416show alternative configurations of the hyperdexterous surgical arm. Thehyperdexterous surgical arm can preserve a Remote Center 250, asdescribed herein. The hyperdexterous surgical arm can be incorporatedinto the hyperdexterous surgical systems described herein. Thehyperdexterous surgical arm can be controlled by input devices 500 whichenable the operator to be mobile.

FIG. 14 shows an embodiment of a hyperdexterous surgical arm 2000. Thehyperdexterous surgical arm 2000 can include a base 2002 including afirst roller slide 2004 and a second roller slide 2006. The first rollerslide 2004 can be orthogonal to the second roller slide 2006. The firstroller slide 2004 permits movement in a first direction (e.g. vertical).The second roller slide 2006 permits movement in a second direction(e.g. horizontal). The rolling slide 2004 can position an offset arm2008 along the direction of Arrow 2010 and/or rolling slide 2006 canposition the offset arm along the direction of Arrow 2012. The firstroller slide 2004 and/or the second roller slide can be passive (e.g.,not motorized). The first roller slide 2004 and/or the second rollerslide can be active (e.g., motorized).

The hyperdexterous surgical arm 2000 includes an offset arm 2008. Theoffset arm 2008 is offset (e.g., the structure of the offset arm 2008 isnot concentric or symmetric about its longitudinal axis). The offset arm2008 includes two degrees of freedom (pitch and roll). The offset arm2008 can include a pitch mechanism 2024 and a roll mechanism. The pitchmechanism 2024 has a pitch axis. The offset arm 2008 can rotate a trocar302 about the pitch axis, about Arrow 2021. The roll mechanism has aroll axis 2018. The offset arm 2008 can rotate around the roll axis 2018in the direction of Arrow 2020. The pitch axis and the roll axisintersect the Remote Center 250. The rolling slide 2004 and the rollingslide 2006 also allow motion only about axes which pass through theRemote Center.

The distal end of the offset arm 2008 may be coupled to a trocar carrier2022. In some embodiments, the offset arm 2008 is coupled to the trocarcarrier 2022 by an arcing slide. An arcing slide having a circularradius will maintain a fixed center of rotation, such as the RemoteCenter 250. The shape of the arc can be altered. In some embodiments,the shape of the arc is elliptical. By alternating the shape, thelocation of Rotation Center could be made to vary in the verticaldirection. In one embodiment, the pitch mechanism 2024 is provided atleast in part by the rolling slide 2004 coupled to the trocar carrier2022.

The rolling slide 2024 may extend up to +/−90° from vertical (including+/15°, +/−30°, +/−45°, +/−60°, +/−75°, etc.) by rolling between thedistal end of the hyperdexterous surgical arm 2000 and the trocarcarrier 2022 in a curved, telescoping fashion. When the rolling slide2024 is vertical, the rolling slide 2024 may be within the profile ofthe trocar carrier 2022. This may minimize the swept volume, therebyreducing interference with surrounding tissue, other equipment, and/oroperating room personnel.

The pitch mechanism 2024, the roll mechanism, first roller slide 2004,and the second roller slide 2006 can be passive or active. In someembodiments, the motion of the hyperdexterous surgical arm 2000 may beactively controlled and may be manipulated by an energy source (e.g.,motors such as electric motors, hydraulics, pneumatics, etc. not shown).

The offset arm 2008 has two degrees of freedom (pitch and roll). Therolling slide 2004 and the rolling slide 2006 can provide two degrees offreedom by providing movement along Arrow 2010 and Arrow 2012. Thehyperdexterous surgical arm 2000 can have four degrees of freedom.

FIG. 15 shows an embodiment of a hyperdexterous surgical arm 2026. Thehyperdexterous surgical arm 2026 may include three segments, segment2028, segment 2030, and segment 2032. The segments 2028, 2030, and 2032are coupled to each other by roll mechanism. In other embodiments, thesegments 2028, 2030, 2032 are coupled to each other via other suitablemechanisms. The three mechanisms have axes of rotation 2034, 2036, 2038.These axes intersect at the Remote Center 250. The third roll mechanismin the hyperdexterous surgical arm 2026 provides a redundant degree offreedom.

The distal end of the hyperdexterous surgical arm 2026 may be coupled toa hyperdexterous surgical tool 2046 and/or a trocar 2048. Thehyperdexterous surgical arm 2026 may include a rotate/translatemechanism 2044 which translates and rotates the tool 2046. The tool 2046may have two degrees of freedom (pitch, yaw). The combination of thehyperdexterous surgical arm 2026, the rotate/translate mechanism 2044,and the hyperdexterous surgical tool 2046 can have seven degrees offreedom. The tool 2046 can have additional degrees of freedom.

FIG. 16 shows an embodiment of an alternative pitch mechanism. The arm2050 of FIG. 16 can be combined with additional mechanism (e.g., rollmechanisms, pitch mechanisms) in order to provide a hyperdexteroussurgical arm. The arm 2050 can preserve a Remote Center 250. The arm2050 can include a roll mechanism that rotates about the roll axis 2058.The follower plate 2054 can rotate about the roll axis 2058 and can besupported by a bearing (not shown). The arm 2050 can be mounted to othermechanisms in such a way that the arm 2050 would rotate about roll axis2058, supported by the bearing (not shown) and actuated by a motor (notshown).

The arm 2050 can provide a pitch mechanism that rotates about the pitchaxis. The pitch mechanism can take a number of forms include four barlinkages, band- or cable-constrained parallel mechanisms, or gear andcam. A gear and cam mechanisms is shown in FIG. 16.

The follower plate 2054 can include a profiled slot 2056. The followerplate 2054 can include a gear profile 2060 which can engage a gearfollower 2062. The gear profile 2060 may be non-circular, and/or thegear follower can be non-circular. The gear follower 2062 can drive acable spool 2064 which can drive a cable 2066. The cable 2066 can becoupled to the arm 2051. The arm 2051 may include gears or pulleys thatinteract with the cable 2066. The cable 2066 drives the rotation of theoutput pulley 2068. The output pulley 2068 may be near the distal end ofthe arm 2051 and may be coupled to the trocar 2070. The arm 2051 mayslide through a linear bearing 2072. The profiles of the profiled slot2056, non-circular gear profile 2060 and non-circular gear follower 2062may cause the trocar 2070 to rotate about the pitch axis. The shape ofthe arm 2051 may be straight or be curved as shown for tissue clearance.

The arm 2050 has one degree of freedom provided by the pitch mechanism.The arm 2050 can have one degree of freedom provided by a roll mechanism(not shown), with a roll axis 2058. Additional redundant degrees offreedom can be added to the arm 2050 to make the arm 2050 ahyperdexterous surgical arm. The hyperdexterous surgical tool (notshown) and the rotate/translate mechanism can provide four degrees offreedom, for a total of six degrees of freedom.

The pitch mechanism and the roll mechanism can be passive or active. Insome embodiments, the motion of the arm 2050 may be actively controlledand may be manipulated by an energy source (e.g., motors such aselectric motors, hydraulics, pneumatics, etc. not shown). The operator 1can mount the arm 2050 to establish the Rotation Center 250.

Hyperdexterous Surgical Tool

The hyperdexterous surgical tool 300 and the rotate/translate mechanism208 provides four degrees of freedom for the hyperdexterous surgicalsystem 100. The rotate/translate mechanism 208 can both rotate andtranslate the hyperdexterous surgical tool 300. The smaller size of therotate/translate mechanism 208 advantageously allows the hyperdexteroussurgical arm 200 to be smaller in size, as discussed previously. Thesmall size of the rotate/translate mechanism 208 enables thehyperdexterous surgical arm 200 to be smaller and lightweight, whichamong other things enables more free space around the patient 2.

The hyperdexterous surgical tool 300 can be rotated and/or translated bya rotate/translate mechanism 208. The rotate/translate mechanism 208rotates the hyperdexterous surgical tool 300 along with therotate/translate mechanism 208. The rotate/translate mechanism 208translates the hyperdexterous surgical tool 300. The rotate/translatemechanism 208 can provide any combination of translation and/orrotation. The rotate/translate mechanism 208 can advantageouslyaccommodate tool shafts of various diameters. The rotate/translatemechanism 208 accommodate tool shafts of any length. The mechanisms thatinteract with the tool shaft as described in FIGS. 17-22 are not limitedto a specific number of rotations. Therefore the mechanism canaccommodate a tool shaft of any length, which is an advantage overexisting, on-market translation mechanisms which use telescopingsegments that are inherently limited in range. The compact size of therotate/translate mechanism 208 can be lighter, allowing for the use of asmaller hyperdexterous surgical arm.

The end effector provides two degrees of freedom (pitch, yaw) and canprovide additional degrees of freedom (jaw actuation, pinch). Referringback to FIG. 10, the hyperdexterous surgical tool 300 includes the endeffector 306, for example a grasper, a needle holder, a stapler, acauterizing tool, deployed at the tip of an elongated shaft. Thehyperdexterous surgical tool 300 can be introduced through a smallincision in the body (e.g., of the patient 2).

FIG. 7 shows the hyperdexterous surgical arm 200 and therotate/translate mechanism 208. The rotate/translate mechanism 208provides two degree of freedom to the hyperdexterous surgical system 100(rotate, translate). Among the degrees of freedom imparted by therotate/translate mechanism are rotation of the hyperdexterous surgicaltool 300 about the tool shaft axis 252 and linear translation of thehyperdexterous surgical tool 300 along the tool shaft axis 252 (see FIG.10). The hyperdexterous surgical tool 300 may be rotated or translatedwithout moving the Remote Center 250. The rotate/translate mechanism 208imparts rotation and/or translation directly onto the hyperdexteroussurgical tool 300 that is supported by the hyperdexterous surgical arm200.

The rotation of the rotate/translate mechanism 208 is transformed intorotation of the hyperdexterous surgical tool 300. The translation of therotate/translate mechanism 208 is transformed into translation of thehyperdexterous surgical tool 300. The direction and speed of thepulleys, geared wheels, or other engagement mechanisms of therotate/translate mechanism 208 results in different types of motion ofthe hyperdexterous surgical tool 300.

The smaller size of the rotate/translate mechanism 208 may ensure thatthat the hyperdexterous surgical arm 200 can be smaller in size. Thesmaller size may reduce the chances for collision with other componentsof the hyperdexterous surgical system 100 (e.g., other hyperdexteroussurgical arms 200, other segments). The smaller size of the proximalsection of the hyperdexterous surgical tools 300 may ensure that thatthe hyperdexterous surgical tool 300 encounters fewer restrictions ofmovement. The smaller weight of hyperdexterous surgical tools 300 allowsthe use of drive mechanisms, such as motors, that are less bulky.Further, the need for large and powerful motors may be reduced.

The small size of the rotate/translate mechanism 208 enables more freespace around the patient 2. The free space enables the surgeon tomanipulate a manual tool 350 simultaneously with the hyperdexteroussurgical tool 300 from various positions. The free space enables thesurgeon to reposition himself at multiple locations during surgery. Thefree space enables the operator to use manual tools 350 concurrentlywith use of the hyperdexterous surgical arm 200. The free space permitseasier physical access to the patient when necessary.

Due to the smaller space taken up by hyperdexterous surgical system 100,the operator 1, such as a surgeon or surgical team member, canadvantageously enable more free space around the patient. The free spaceenables the operator 1 to readily gain access to the patient.

FIGS. 17-22 illustrate one embodiment of the rotate/translate mechanism208. Two different types of rotate/translate mechanisms 208 will bediscussed below—the asymmetric rotate/translate mechanism 258 and asymmetric rotate/translate mechanism 2500. There may be other types ofrotate/translate mechanisms 208 as well. The rotate/translate mechanism258, 2500 may use rollers, gears, pulleys, friction surfaces, etc. ineither symmetric or asymmetric differential configurations. It may alsobe combined with the hyperdexterous surgical tool in order to minimizethe number of discrete parts of the system and increase the ease of useand setup. The asymmetric rotate/translate mechanism 258 can include atleast two pulleys, a pulley 260 and a pulley 262. The asymmetricrotate/translate mechanism 258 has a central housing 264 located throughthe centers of the pulleys 260, 262. The pulley 260 can be coupled tothe central housing 264. The pulley 262 can rotate about the centralhousing 264. The hyperdexterous surgical tools 300 can be insertedthrough the central housing 264.

The pulley 260 can include at least two rollers, a roller 266 and aroller 268. Referring now to FIG. 18, the roller 266 has a roller wheel270 and a roller drum 272. The roller 268 has a roller wheel 274 and aroller drum 276. The diameter of the roller drums 272, 276 may besmaller than the roller wheel 270, 274. The roller wheel 270 has aconcave surface 278 and the roller wheel 274 has a concave surface 280.The concave surfaces 278, 280 can conform to the shape of thehyperdexterous surgical tool 300 and facilitate the ability of therollers 266, 268 to grip or engage the hyperdexterous surgical tool 300through slots in the central housing 264. Although roller wheel 270 androller wheel 274 are shown to have concave surfaces, various othersurfaces may be utilized to engage the surface of the tool shaft of thehyperdexterous surgical tool 300, including but not limited to texturedsurfaces, gear teeth, and belts.

The pulley 260 includes at least two additional rollers, a roller 282and a roller 284. The rollers 282, 284 are attached to the underside ofthe pulley 260. FIG. 17 shows the roller 282 attached to the undersideof the pulley 260.

The process of translation of the hyperdexterous surgical tool 300 isshown in FIGS. 19 and 20. As shown in FIG. 19, a motor 286 drives thepulley 260 and a motor 288 drives pulley 262. As mentioned previously,the hyperdexterous surgical tool 300 can be inserted through the centralhousing 264. If the motor 288 is driven such that the pulley 262 rotatesin the direction of Arrow 1, the roller 282 will rotate in the directionof Arrow 2. Turning to FIG. 20, which is the top view illustrating thesame motion as shown in FIG. 19, the roller 282 is shown rotating in thedirection of Arrow 2.

The roller 284 is not shown in FIG. 19. Turning to FIG. 20, which is thetop view illustrating the same motion as shown in FIG. 19; the roller284 is shown rotating in the direction of Arrow 3. The rotation of therollers 282, 284 will cause the roller drum 272 and the roller drum 276to rotate. The rotation of the roller drums 272, 276 will cause theroller wheel 270 and the roller wheel 274 to rotate. The rotation of theroller 266, including the roller wheel 270 and the roller drum 272, isshown by Arrow 4. The rotation of the roller 268, including the rollerwheel 274 and the roller drum 276, is shown by Arrow 5.

The rotation of the roller wheel 270 of roller 266 and the rotation ofthe roller wheel 274 of roller 268 causes the hyperdexterous surgicaltool 300 inserted in the central housing 264 to translate. However, thehyperdexterous surgical tool 300 must be allowed to uncouple from thecentral housing 264 to permit the use of a different hyperdexteroussurgical tool 300. The hyperdexterous surgical tool 300 translates inthe direction of Arrow 6, shown in FIG. 19. Arrow 6 is not shown in FIG.20 as it would be perpendicular, out of the plane of the drawing. Totranslate the hyperdexterous surgical tool 300 in the downwarddirection, the motor 288 simply turns in the reverse direction, reverseto Arrow 1. This causes the motions of the rollers 266, 268, includingthe roller wheels 270, 274 and the roller drums 272, 276, to rotate inreverse. This causes the hyperdexterous surgical tool 300 to translatein the opposite direction to the direction shown in FIGS. 19 and 20.

The process of rotation of the hyperdexterous surgical tool 300 is shownin FIGS. 21 and 22. In FIG. 21, the pulley 260 and the pulley 262 arerotated by the motor 286 and the motor 288, respectively. Both pulleys260, 262 are rotated in the same direction, shown by Arrow 7. Due to themotion of both motors 286, 288, the roller 282 will not rotate in thedirection of Arrow 2, as shown in FIGS. 19 and 20. Due to the motion ofboth motors 286, 288, the roller 284 will not rotate in the direction ofArrow 3, as shown in FIG. 20. When both pulleys 260, 262 rotate in thesame direction, the roller 282 and 284 do not rotate about their ownrotation axes 290, 292. Subsequently, the rollers 266 and 268 do notrotate about their own rotation axes 294, 296, as shown in FIG. 20. Therollers 266, 268, 282, 284 do rotate with the pulley 260, as shown byArrow 8 in FIG. 22. The pulley 260 rotates about the central housing264. As the pulley 260 is rotated, the hyperdexterous surgical tool 300rotates as shown by Arrow 9. To rotate in the other direction, themotors simply turns in the reverse direction, reverse to Arrow 7 shownin FIG. 21.

The rotate translate 258 uses engagement mechanisms such as rollers andfriction wheels. Other types of engagement mechanism may additionally oralternatively be utilized including gears, belts, beveled gears, andcables.

FIGS. 23-27 illustrate another embodiment of a rotate/translatemechanism 2500. This mechanism falls in the category of symmetricrotate/translate and differential rotate/translate mechanisms. Thesymmetric rotate/translate mechanism 2500 can include a beveled gear2504 and a beveled gear 2506. In FIG. 23, the central housing 2502 isgenerally perpendicular to the beveled gears 2504, 2506. The beveledgears 2504, 2506 include teeth, which are shown in FIG. 24.

The motor 2508 (e.g., electric motor) includes motor gear 2510. Themotor 2508 and motor gear 2510 drive beveled gear 2504. The beveled gear2504 drives an inset beveled gear 2512. The inset gear 2512 and theinset gear 2526 are connected to the central housing 2502 but can rotateabout their own central rotation axes. The inset beveled gear 2512 iscoupled with the central housing 2502. The inset beveled gear 2512rotates about the axle 2514. The axle 2514 is coupled to a secondarygear 2516. The secondary gear 2516 interfaces with a spur gear 2518. Thespur gear 2518 is connected to a roller 2520.

The motor 2522 (e.g., electric motor) includes motor gear 2524. Themotor 2522 and motor gear 2524 drive beveled gear 2506. The beveled gear2506 drives an inset beveled gear 2526. The inset beveled gear 2526 iscoupled with the central housing 2502. The inset beveled gear 2526rotates about the axle 2528. The axle 2528 is coupled to a secondarygear 2530. The secondary gear 2530 interfaces with a spur gear 2532. Thespur gear 2532 is connected with a roller 2536.

FIG. 24 shows the rollers 2520, 2536 and the beveled gears 2504, 2506.The symmetric rotate/translate mechanism 2500 includes a linear drivebelt 2538 coupled with the roller 2520 and a linear drive belt 2540coupled with the roller 2536. The roller 2520 is placed between thelinear drive belt 2538 and the central housing 2502, so that the lineardrive belt 2538 partially wraps around the roller 2520. The roller 2536is placed between the linear drive belt 2540 and the central housing2502, so that the linear drive belt 2540 partially wraps around theroller 2536. The linear drive belt 2538 and the linear drive belt 2540can extend at least partially along the length of the hyperdexteroussurgical tool 300 and are coupled to the shaft of the hyperdexteroussurgical tool 300.

The process of translation of the hyperdexterous surgical tool 300 isshown in FIGS. 25 and 26. The motor 2522 is rotated as shown in thedirection of Arrow 1. This causes the beveled gear 2506 to rotate in thedirection of Arrow 2. The beveled gear 2506 causes the inset beveledgear 2526 to rotate in the direction of Arrow 3. The rotation of insetbeveled gear 2526 causes the secondary gear 2530 rotate in the samedirection as Arrow 4. The rotation of the secondary gear causes 2530 thespur gear 2532 to rotate in the direction of Arrow 5. The rotation ofthe spur gear 2532 causes the roller 2536 to rotate in the direction ofArrow 6. Referring now to FIG. 26, the roller 2536 rotates in thedirection of Arrow 6.

The motor 2508 is rotated as shown in the direction of Arrow 7. Thiscauses the beveled gear 2504 to rotate in the direction of Arrow 8. Thebeveled gear 2504 causes the inset beveled gear 2512 to rotate in thedirection of Arrow 9. The rotation of inset beveled gear 2512 causes thesecondary gear 2516 rotate in the direction as Arrow 10. The rotation ofthe secondary gear 2516 causes the spur gear 2518 to rotate in thedirection of Arrow 11. The rotation of the spur gear 2518 causes theroller 2520 to rotate in the direction of Arrow 12. Referring now toFIG. 26, the roller 2520 rotates in the direction of Arrow 12. Therollers 2520 engage opposite sides of the hyperdexterous surgical tool300, as shown in FIG. 26. In FIGS. 23 and 25, the roller 2520 is behindthe central housing 2502 and the hyperdexterous surgical tool 300 and istherefore shown in dashed lines.

The rotation of the rollers 2520, 2536 will cause linear translation ofthe linear belt drives 2538, 2540 in the direction indicated by Arrow 13and Arrow 14 in FIG. 26. The motion of the linear belt drives 2538, 2540causes linear translation of the hyperdexterous surgical tool 300. Thelinear belt drives 2538, 2540 are coupled to the tool shaft of thehyperdexterous surgical tool 300, and the hyperdexterous surgical tool300 translates through the central housing 2502. A hyperdexteroussurgical tool 300 inserted in the central housing 2502 would translatein the direction of Arrow 13 and Arrow 14. To move the hyperdexteroussurgical tool 300 in the upward direction, the motors 2508, 2522 wouldsimply rotate in the opposite direction, a direction opposite to Arrow 1and Arrow 7 in FIG. 25.

The process of rotation of the hyperdexterous surgical tool 300 is shownin FIGS. 27 and 28. The motors, motor gears, secondary gears, spur gearsand rollers below the linear belt drive are not shown. In addition, theteeth for the beveled gears and the inset gears are not shown. In FIG.27, the beveled gears 2504, 2506 are rotated by the motors (not shown).Both beveled gears 2504, 2506 are rotated in the same direction, shownby Arrow 15. Due to the motion of both motors, the inset beveled gears2512, 2526 will not rotate in the direction of Arrow 3 and 9, as shownin FIG. 22.

Due to the motion of both motors, the rollers 2520, 2536 rotate with thebeveled gears 2504, 2506, as shown by Arrow 16 in FIG. 27. The rotationof both bevel gears 2504, 2506 in the same direction causes the centralhousing 2502 and the captured hyperdexterous surgical tool 300 to allrotate at the same speed. As the beveled gears 2504, 2506 are rotated,the hyperdexterous surgical tool 300 rotates as shown by Arrow 16. Torotate in the other direction, the motors simply turn in the reversedirection.

Due to the motion of both motors 2508, 2522, the inset beveled gears2512, 2526 rotate with the beveled gears 2504, 2506. The inset beveledgears 2512, 2526 are coupled with the central housing 2502. The rotationof the bevel gears 2504, 2506, and the inset beveled gears 2512, 2526 inthe same direction causes the central housing 2502 and the capturedhyperdexterous surgical tool 300 to all rotate at the same speed. Theengagement mechanisms such as gears, belts, and beveled gears can beutilized in the rotate/translate mechanism 208. Other types ofengagement mechanism may additionally or alternatively be utilized inother embodiments such as rollers, bearings, and cables.

FIG. 28 illustrates an alternative embodiment of the linear belt drives2538, 2540 and rollers 2520, 2536 of FIGS. 23-27. The embodimentincludes smaller continuous belt drives 2546, 2548. The belt drive 2546surrounds two rollers 2550, 2552. The belt drive 2548 surrounds tworollers 2556, 2558. The teeth 2560, 2562 for the belt drives 2546, 2548are placed on the outside surface of the belt drives 2546, 2548. Theteeth 2560, 2562 for the belt drives 2546, 2548 engage the teeth 2564,2566 on the outside surface of the central housing 2502 or directly onthe tool shaft of the hyperdexterous surgical tool 300. Although beltdrives are used to engage the central housing 2502 or the tool shaft ofthe hyperdexterous surgical tool 300 in the illustrated embodiment,various other suitable mechanisms may be utilized to engage the centralhousing 2502 or the tool shaft of the hyperdexterous surgical tool 300.

The rotate/translate mechanism 208 can provide any combination oftranslation or rotation. Referring back to FIG. 23, assume that theanticlockwise rotation of the motor 2508 and the anticlockwise rotationof the motor 2522 is assigned a “+” direction and the clockwise rotationof motor 2508 and 2522 is assigned a “−” direction. Further also assumethat one unit of translation of the central housing 2502 or the toolshaft of the hyperdexterous surgical tool 300 in the downward directionis called “T+” and one unit of clockwise rotation of the central housing2502 is called “R+”. This nomenclature may be reversed without changingthe concept. It must be further noted that the one translation unit maynot correspond with an integral units of distance, such as 1 cm.Similarly one rotation unit may not correspond to a complete rotation ofthe central housing 2502. The conversion of these units to actualdistance or degrees depends on the gear ratio selected.

Assuming that one unit of rotation of the motor 2508 or “MA+” producesone translation unit in the positive direction and one rotation unit ofthe central housing 2502 in the positive direction. Thus:

MA+=T++R+  Eqn. 1.

Similarly, one unit of rotation of the motor 2522 or MB+ produces onetranslation unit in the positive direction and one rotation unit of thecentral housing 2502 in the negative direction. Thus:

MB+=T++R ⁻  Eqn. 2.

For example, to get two units of translation in the positive direction:

MA++MB+=2T+  Eqn. 3.

To get one unit of rotation in the negative direction

½MA ⁻+½MB+=R ⁻  Eqn. 4.

Thus any combination of translation and/or rotation may be obtained bycombining the motions of the motors 2508, 2522 including the combinationwhere one motor does not move. In some embodiments, the speed oftranslation and rotation may be varied using the same concepts. Thespeed of the motors 2508, 2522 may be set as a predetermined value. Insome embodiments, through the display 600, an operator 1 (e.g., surgeon)may want to choose a slower, more sensitive setting, whereas anotheroperator may prefer a less sensitive setting. This is analogous to asetting the sensitivity settings of a computer mouse where the responseof the pointer on the screen may follow the user settings based onpreference settings.

In some embodiments, the translation unit and the rotation unit maycorrespond to actual distances and actual degrees of rotations. Theconversion values may be determined during the design stage of thehyperdexterous surgical arm 200 or may be determined via a calibrationprocedure. The conversion values may be stored in memory within thecontrol system 400. Due to play in the components and other factors,each motor 2508, 2522 may not result in exactly the same amount oftranslation and rotation. In this case, these differences may beaccounted by the control system 400 and in the equations describedabove.

The rotate/translate mechanism 208 can advantageously accommodate toolshafts of various diameters. FIGS. 29-31 show a width adjuster 2600coupled to the asymmetric rotate/translate mechanism 258, describedherein. The width adjuster 2600 can be coupled to the symmetricalrotate/translate mechanism 2500. FIG. 29 shows a side view of the widthadjuster 2600, coupled to the central housing 264. The width adjuster2600 may move up and down along the length of the central housing 264.The width adjuster can have a plurality of links 2602A, 2602B, 2602C(not shown), 2602D (not shown) attached to the rollers 266, 268, and282. A link can be attached to roller 284 (not shown). The links 2602A,2602B may be arranged in a triangular shape. The links 2602A connect thewidth adjuster 2600 to the roller 266 and the roller 282 in a generallytriangular shape. The links 2602B connect the width adjuster 2600 to theroller 268 and the roller 282 in a generally triangular shape. The links2602C connect the width adjuster 2600 to the roller 266 and the roller284 in a generally triangular shape (not shown). The links 2602D connectthe width adjuster 2600 to the roller 268 and the roller 284 in agenerally triangular shape (not shown). The triangle shape is an exampleof shapes that can be deployed, and other shapes are possible. The links2602 can be solid pieces.

The links 2602 pivot around the axis of rotation of the rollers 282 and284. The links 2602 can adjust the position of the centers of roller 266and 268. The rollers 266, 268 have shafts that extend along theirrotation axis 294, 296. The shafts of the rollers 266 and 268 arecoupled to the links 2602.

The width adjuster 2600 can translate along the axis of the centralhousing 264. When the width adjuster 2600 moves downward, the links 2602pivot to assume a new position. The links pivot around the axes 290, 292of the rollers 282, 284. In this new position of the links 2602, therollers 266, 268 assume a new position, as shown in FIG. 31. The newposition of rollers 266′, 268′ permit a hyperdexterous surgical tool 300of a greater diameter to be inserted into the central housing 264. Asshown in FIGS. 30-31, the old position of the rollers 266, 268 is shownin dashed lines, while the new position of the rollers 266′, 268′ isshown is shown in solid lines. There is more space between the roller inthe new position 266′ and 268′.

In another example, in obese patients the manipulation of tools becomesdifficult due to the body habitus; in these cases a larger or longerhyperdexterous surgical tool 300 can be supported by the hyperdexteroussurgical arms 200. The width adjuster 2600 enables larger diameter toolsto be used. The rotate/translate mechanism 208 enables the uses oflonger tools.

In other aspect of the invention, it is often desirable to know how fara hyperdexterous surgical tool 300 is inserted into the body of apatient 2. In a surgical setting, the hyperdexterous surgical tool 300is inserted into the body of the patient 2, such as the abdomen of thepatient 2. It may be advantageous to know how much of the hyperdexteroussurgical tool 300 is inside the body of the patient 2 and/or know howmuch of the hyperdexterous surgical tool 300 is outside the body. Thisdistance may be determined by monitoring and calculating electricalparameters such as resistance or capacitance of the section of thehyperdexterous surgical tool 300 that is outside the body of the patient2.

A contact between the body of the patient 2 and the hyperdexteroussurgical tool 300 is made at the point of entry. An electrical circuitmay be completed between a set point on the hyperdexterous surgical tool300 outside the patient and the contact point on the hyperdexteroussurgical tool 300 at the point of entry. As the hyperdexterous surgicaltool 300 is manipulated, electrical parameters such as resistance orcapacitance between the set point and the contact point may change. Thischange may be monitored and further processed (e.g., by the controlsystem 400) to calculate the ratio of the tool outside the patient 2 tothe ratio of the tool inside the patient 2.

Input Devices

The input devices 500 control the hyperdexterous surgical arm 200 and/orthe hyperdexterous surgical tools 300. The input devices 500 can bewireless and/or portable allowing the operator to move about the patient2. The input devices 500 allow the operator to be at various locationswithin the operating arena (e.g., various locations at the patient'sbedside). The input devices 500 enable the operator to control thehyperdexterous surgical tools 300 and the manual tool 350simultaneously. The simultaneous manipulation the hyperdexteroussurgical tool 300 and the manual tools 350 advantageously reduces theneed for a surgical assistant to manipulate the manual tools 350 whilethe operator controls the hyperdexterous surgical tool 300. The inputdevice 500 of the present invention can take a number of forms includingthe pincher 502 (see FIG. 32A) and the controller 514 (see FIG. 2).

The input devices 500 can be wireless or wired. In some embodiments, theinput devices 500 are handheld, portable devices that can be carried bythe operator allowing the operator to move about the patient 2 andoperate the input devices 500 from various locations (e.g., variousbedside locations) during use of the hyperdexterous surgical system 100.The operator 1 can therefore move around the operating arena and controlthe hyperdexterous surgical arm 200 from various locations, asillustrated in FIGS. 3A-3C.

In some embodiments, the input device 500 can be coupled to the body ofthe operator 1. FIG. 32A shows the pincher 502. The pincher 502 mayassist the operator 1 in controlling a hyperdexterous surgical tool 300in a manner consistent with holding the hyperdexterous surgical tool 300with the thumb and the forefinger. The pincher 502 may assist theoperator 1 in controlling a hyperdexterous surgical tool 300 at acontrol point such as the tool tip.

The operator 1, such as a surgeon, wears the two rings 504, 506 of thepincher 502. One ring 504 can be placed around the thumb of the operator1 and one ring 506 can be placed around the forefinger of the operator1. However, the rings 504, 506 can be worn on other fingers based on theconfiguration of the pinchers. There may be an association with themotion of the input device 502 and the motion of a tool tip or theend-effector 306.

The pincher 502 may have various sensors 508 such as position sensorsand gyroscopes. Additional sensors 510 such as strain sensors may alsomeasure the distance between the two arms of the pincher 502. Thesensors 508, 510 can provide information related to the position andorientation of the pincher 502 to the control system 400. Theinformation from these sensors 508, 510 serve as inputs to the controlalgorithms such that the hand, wrist and finger movement of the operator1 may be translated to motion of the control point. The movements of theinput device 502 can control the movements of the hyperdexteroussurgical arm 200, the hyperdexterous surgical tool 300, and/or theend-effector 306 of the hyperdexterous surgical tool 300, such as agripper. In one embodiment, the pincher 502 can include a power sourcesuch as a button cell (not shown).

In some embodiments, a second a pair of rings can be provided. Thesecond pair of rings can be worn on other fingers of the same hand. Theoperator 1 may wear the second pair of rings around other fingers sothat various motions of the hand may be translated to specific commands.The second pair of rings can be worn on different fingers of the samehand controlling the pincher 502. One ring of the second pair of ringscan be placed around the thumb of the operator 1 and one ring of thesecond pair of rings can be placed around the last finger of theoperator 1. As a non-limiting example, the first pair of rings 504, 506of the first pincher 502 can control a first hyperdexterous surgicaltool 300 and the second pair of rings can control a secondhyperdexterous surgical tool 300. As a non-limiting example, the firstpair of rings 504, 506 of the first pincher 502 and the second pair ofrings can control the same hyperdexterous surgical tool 300.

In some embodiments, the second pair of rings and/or a second pincher502 can be worn on the other hand. The second pair of rings and/or asecond pincher 502 can be worn on a different hand of the operator 1than the hand controlling the first pincher 502. As a non-limitingexample, the first pair of rings 504, 506 of the first pincher 502 cancontrol a first hyperdexterous surgical tool 300 and the second pair ofrings and/or the second pincher 502 can control a second hyperdexteroussurgical tool 300. As a non-limiting example, the first pair of rings504, 506 of the first pincher 502 and the second pair of rings and/orthe second pincher 502 can control the same hyperdexterous surgical tool300 (e.g., at different control point).

The movement of the pincher 502 described herein can cause the movementof any control point. The control point can control the end effector(e.g., a grasper motion). However, the input devices 500 need not onlycontrol the end effector 306 or tip of a hyperdexterous surgical tool300. The input devices 500 can control any part of the hyperdexteroussurgical tool 300 by assigning a control point any location along thehyperdexterous surgical tool 300 (e.g., via the display 600). Thecontrol point can be considered a hypothetical location from which theoperator 1 is controlling the tool (e.g., a fulcrum). The control pointcan be created anywhere along the hyperdexterous surgical tool 300, thehyperdexterous surgical arm 200, or any component of the hyperdexteroussurgical system 100.

In some embodiments, the input device 500 is held by the operator 1.FIG. 32B shows another embodiment of an input device 500. In theillustrated embodiment, the input device 500 can be a knob 503. Theoperator 1, such as a surgeon, can hold the knob 503 with the one ormore fingers of a hand. The knob 503 can have one or more buttons 505. Acontrol point of the hyperdexterous surgical tool 300 can be moved oractuated when the operator squeezes at least one of the one or morebuttons 505.

In some embodiments, the input device 500 is fixed relative to theoperator 1. One embodiment is shown in FIG. 2. The hyperdexteroussurgical system 100 can include a platform 602. The input device 500 canbe coupled to the platform 602. The input device 500 can be coupled toany fixture within the operating arena. The input device 500 can takethe form of a controller 514. The controller 514 can be wired orwireless. The user interface sub-system 605 allows an operator 1, suchas a surgeon, to control the wired controller 514 in close proximity tothe display 600, as shown in FIG. 2. The wired controller 514 can belocated below this display 600. The platform 602 may also include ahorizontal resting bar 603.

For example, if the operator 1 during the course of the surgery decidesto sit and control the hyperdexterous surgical tools 300, the controller514 would allow him or her to do that. For certain surgical procedures,the operator 1 may find it more comfortable to move a fixed input devicesuch as the wired controller 514 rather than an input device 500attached to the operator's body such as pincher 502. For instance, theoperator 1 may be able to control his or her movements better whileresting against the horizontal resting bar 602 and/or sitting. Thenatural body position of the operator may be to rest a portion of his orher body against the horizontal resting bar 603 while controlling thewired controller 514. The operator 1 can control other input devices 500(e.g., pincher 502) while resting against the horizontal resting bar603.

The hyperdexterous surgical system 100 can be controlled from multiplelocations in the surgical arena. In some embodiments, the operator 1 cansit or stand while holding or otherwise controlling one or more inputdevices 500 (e.g., pincher 502, wired controller 514). The operator cansupport a portion of his or her body on the resting bar 603. The variousinput devices 500 allow the operator 1 to move around and place him orherself in the most optimal position.

The input device 500 controls the movement of one or more controlpoints. The control points are locations which have the capability toexecute some motion. Control points can be located on the hyperdexteroussurgical arm 200, hyperdexterous surgical tools 300, or any otherlocation. The hyperdexterous surgical tools 300 can be controlled by theinput devices 500 relative to one or more control points 2600. Thecontrol points 2600 are locations on the hyperdexterous surgical arm 200and/or the hyperdexterous surgical tool 300 which have the ability tomove. The input by the operator 1 effects the movement of the one ormore sections that are connected to the control point 2600.

The operator 1 can associate a control point 2600 with the input device500 being manipulated. Moving the input device 500 causes thehyperdexterous surgical arm 200 and/or the hyperdexterous surgical tool300 to move about the control point. The movement of the input device500 would cause movement of the selected control point 2600. This wouldcause movement of the hyperdexterous surgical tool 300 that iscontrolled by the hyperdexterous surgical arm 200. The display 600 maybe used to assign an input device 500 to a specific control point 2600.

The hyperdexterous surgical tools 300 can be controlled by the inputdevices 500 relative to one or more virtual grips 512. The virtual grip512 can increase the flexibility with which the hyperdexterous surgicaltools 300 are positioned within the patient 2. FIGS. 33A and 33B show ahyperdexterous surgical tool 300. In FIG. 33A, the virtual grip 512 isplaced at the end of the hyperdexterous surgical tool 300. If thehyperdexterous surgical tool 300 were to be constrained to move aboutthe location of the virtual grip 512, then the tool tip and end effector306 could only swing a small distance. In FIG. 33B, the virtual grip 512is placed towards the middle of the hyperdexterous surgical tool 300.The tool tip and end effector 306 can swing a larger distance. Thevirtual grip 512 allows the operator 1 such as a surgeon to decidebetween fine motion and coarse motion. By moving the virtual grip 512toward the tool tip, the operator can complete finer maneuvers. Ifmultiple hyperdexterous surgical tools 300 are used, then each tool mayhave a different virtual grip 512. Further, different virtual grips 512can be activated by different input devices 500, such that each virtualgrip 512 is independently controlled. For example, the left hand withone input device 500 may control a hyperdexterous surgical tool 300 witha virtual grip near the end-effector, as shown in FIG. 33A. The righthand with another input device 500 may control the same hyperdexteroussurgical tool 300 with the virtual grip 512 placed in the middle of thehyperdexterous surgical tool 300, as shown in FIG. 33B.

As described herein, the operator 1 may control a hyperdexteroussurgical tool 300 by associating a virtual grip 512 to the tool tip.With his or her other hand, the operator 1 may associate a virtual grip512 to the proximal end of the hyperdexterous surgical tool 300 (e.g.,the same hyperdexterous surgical tool 300 or a different hyperdexteroussurgical tool 300). The virtual grip at the proximal end would allow theoperator 1 to control the hyperdexterous surgical tool 300 similar tohow he or she controls a laparoscopic tool. The natural control of thehyperdexterous surgical tool 300 in this manner may be accomplished byassociating independent frames of reference to each of the operator'shands, as described herein.

The position of the input device 500 in space may be tracked. In someembodiments, the position is tracked by coupling absolute encoders (notshown) to the input device 500. In some embodiments, a position sensor(optical tracker) is mounted at the base of the input device 500. Insome embodiments, the input device 500 is tracked by a sensor that theoperator 1 wears. In some embodiments, the input device 500 is trackedby a sensor on the platform 602. The position sensor can provide theposition of the input device 500 relative to a ground reference point(not shown). The position sensor and/or the encoders can be utilized totrack the position of the input device 500. One skilled in the art mayutilize others suitable sensors, mechanism or methods of trackingcomponents of the hyperdexterous surgical system 100. In someembodiments, more than one (e.g., a plurality, several) trackingtechnologies are used in conjunction. The redundant trackingtechnologies can account for occlusions and detect malfunctions. Theredundant tracking technologies can increase resolution or bandwidth.

In some embodiments, the input devices 500 are sterile or capable ofbeing sterilized. The operator 1 needs to maintain his hands in asterile environment during the course of the procedure. During surgery,the operator 1 may manipulate one or more manual tools 350 and one ormore input devices 500. The input devices 500 can be used in conjunctionwith touching the patient. For instance, the operator 1 can control theinput devices 500 and touch the patient 2 simultaneously (e.g., withtheir hand or a manual tool). The input devices 500 must be sterile orcapable of being sterilized to maintain the sterile operatingenvironment.

In some embodiments, the input devices 500 can include one or morefeatures that interact with the control system 400. For instance, theinput devices 500 can control a camera 304 (see FIG. 35), such as acamera placed in the workspace inside the patient or in the free spaceabove a patient. The camera 304 can be considered a hyperdexteroussurgical tool 300 and controlled by an input device 500. The inputdevices 500 can alter the images shown on the display 600, 702. Imagesmay be inverted, rotated, and left-right flipped on the display 600, 702to reflect the viewpoint of the tracked objects, such as the operator 1,or controllable objects such as hyperdexterous surgical arms 200 and/orhyperdexterous surgical tools 300, as described below. In someembodiments, the input devices 500 include a button and/or a slider. Thebutton and/or slider can be actuated by the operator 1 to change thecamera 304 pan/tilt and/or zoom. In some embodiments, the operator 1depresses a button on the input device 500 and uses motions of anotherpart of the operator's body to change the camera parameter. For example,the operator 1 can depress the button and use motions of his eyes tochange a camera parameter. In some embodiments, the operator 1 moves theslider (e.g., with a finger) to change the zoom of the camera 304. Insome embodiments, the input devices 500 can accept information from thecontrol system 400, described below. The control system can sendinformation to the input devices 500, such as instructions to producetactile feedback for the operator 1. The tactile feedback can be sentvia a wireless or a wired connection. The input device 500 can integratea sensor which imitates the tactile feedback of using a tool. The inputdevice 500 can imitate the tactile feedback of the hyperdexteroussurgical tool 300. The input device 500 can imitate the tactile feedbackof the manual tool 350. The hyperdexterous surgical system 100 can linkthe actual tactile feedback of the manual tool 350 with an imitationtactile feedback conveyed by the input device 500. In some embodiments,the input device 500 can be docked at an interface (e.g., platform 602)which can provide tactile feedback.

Control System

The hyperdexterous surgical system 100 can include a control system 400that translates user inputs (e.g., via input devices 500, display 600)to outputs (e.g., motion of control points, images). The control system400 can pair certain user inputs to certain controllable objects. Whenexecuting the movements of the various hyperdexterous surgical tools300, the control system 400 may maintain one or more constraints. Thecontrol system 400 can lock one or more hyperdexterous surgical tools300 to a single tool (e.g., a hyperdexterous surgical tool 300 or amanual tool 350). Advantageously, the control system 400 allows thehyperdexterous surgical system 100 to enable the flexibility provided tothe operator (e.g., surgeon) for performing surgical procedures. Forexample, as discussed in more detail below, the control system 400allows the surgeon to control the hyperdexterous surgical tools 300using various frames of reference (e.g., immersive camera mounted frameof reference, world-grounded frame of reference), and the ability tomove between various frames of reference, which advantageously allowsthe surgeon to move seamlessly between operating the hyperdexteroussurgical tools 300 with a camera-mounted frame of reference to perform asurgical task, to a world-grounded frame of reference that allows thesurgeon to reposition him or herself in another position while retainingawareness of the position of the hyperdexterous surgical tools 300relative to himself, and then moving back to a camera-mounted frame ofreference to continue with a surgical task to begin a different surgicaltask. Additionally, the control system 400 advantageously allows thesurgeon to switch control of hyperdexterous surgical tools 300 (e.g.,between right and left), to allow repositioning of the surgeon to anoptimal position, and to rotate the camera view accordingly. Further,the control system 400 can advantageously allow one or morehyperdexterous surgical tools 300 to be controlled so that it moves insynchrony with another tool (e.g., with another hyperdexterous surgicaltool 300, or with a manual tool 350), which can allow the surgeon tovirtually tether the tools and move them at the same time, such as whenmoving the tools to another surgical site.

FIG. 34 illustrates an embodiment of the control system 400. The controlsystem 400 architecture may be thought of as containing three sections;a section 408 which receives inputs, a section 410 responsible forsending output, and a section 412 that computes the outputs based onvarious data including the inputs. The operator 1 can provide an inputvia the input device 500. The inputs can be provided by a wireless orwired input device 500. The location of the operator 1 can be an input(e.g., as tracked by a tracking device that communicates with thecontrol system 400). Additional components can provide inputs includingthe display 600 and the clutch 112.

The control system 400 may receive inputs from location and orientationsensors of the hyperdexterous surgical arm 200, the hyperdexteroussurgical tool 300 and/or the manual tool 350. The control system 400computes outputs based in part on the various inputs. The outputs maymanipulate the one or more hyperdexterous surgical tools 300 and/or theone or more hyperdexterous surgical arms 200. The outputs may be tactilesignals sent to the input device 500 to be felt by the surgeon. Theoutput may be images shown on the one or more displays 600, 702.

FIG. 35 shows an embodiment of the control system 400 with the computer402. The control system 400 can be coupled with several controllableobjects, such as one or more hyperdexterous surgical arms 200 and one ormore hyperdexterous surgical tools 300.

The control system 400 can be coupled with several input devices 500.The arrows indicate the flow of information between the control system400 and the input devices 500. The control system can send informationto the input devices 500, for instance tactile feedback as shown by thearrow. The dotted arrow is meant to indicate a wireless communication.The double arrows indicate an input to the control system and an outputfrom the control system. The tactile feedback can be sent via a wirelessor a wired connection. The input devices 500 can be used to control themovement of one or more hyperdexterous surgical arms 200 and/or one ormore hyperdexterous surgical tools 300.

Other than input devices 500, the control system maybe coupled toseveral output devices as well. The control system 400 can be coupledwith several devices, such as the display 600 and the display 702. Thedisplay 600 can also be an input. The display 600 provides informationto the operator 1 and accepts information from the operator 1. Thedisplay 600 can be a touch screen monitor 604. Other types of displays600 may also be used such as an iPad or other mobile electronic device.In some embodiments, the display 600 is sterile or capable of beingsterilized. The display 600 can be used in conjunction with manual tools350. The display 600 can be used in conjunction with touching thepatient. The operator 1 can control the display 600 and manual tools 350simultaneously. The operator can control the display 600 and touch thepatient 2 simultaneously.

The control system 400 can be coupled with several controllable objects,such as hyperdexterous surgical arms 200 and/or hyperdexterous surgicaltools 300. The control system 400 can be coupled with several inputdevices 500. The double headed arrow indicates the flow of informationbetween the computer 402 and the input devices 500. The clutch 112 canprovide an input to the computer.

The manual tool 350 (e.g., a sensor 352 affixed to a manual tool 350)can provide an input to the computer. As described earlier manual tools350 can also be used with the hyperdexterous surgical system 100. Thesensors 352 on the manual tool 350 may provide an input to the computer402. The manual tool 350 can be tracked by the sensors 352, such asposition sensors. The sensors 352 can be affixed, for example, by acollar or a sleeve that fits over the manual tool 350. The collar orsleeve can include various sensors 352, which may be wireless or wired.In some embodiments, the sensors 352 may be integrally formed with themanual tool 350. The tracking of manual tools 350 is useful in manysituations, such as when the manual tool 350 is not within the range ofsight of the camera 304.

The display 600 may show the associations (e.g., pairings) between theinput devices 500 and the controlled objects, such as the one or morehyperdexterous surgical arms 200 and/or the one or more hyperdexteroussurgical tools 300. As a non-limiting example, the display 600 may showicons for the input devices 500 and the controlled objects, such as theone or more hyperdexterous surgical arms 200 and/or the one or morehyperdexterous surgical tools 300.

FIG. 36 shows one embodiment of a screen shot of a display 600. In thisexample, the operator 1 selects an input device 500. As shown in FIG.36, two input devices 500 are available to be selected by the operator1. These input devices 500 are two wireless controllers 514, but otherinput devices 500 can be depicted on the display 600 (e.g., wiredcontrollers). As shown in FIG. 36, five controllable objects areavailable to be selected by the operator. These controllable objects arefive hyperdexterous surgical tools 300 including one camera 304. Thecamera 304 can be considered a hyperdexterous surgical tool 300. Othercontrollable objects can be depicted on the display 600. The operator 1can select one of the input device icons. The operator 1 can select oneof the controllable object icons. In some embodiments, the operator 1selects an input device icon and then selects a controllable object iconin sequence, which will cause the selected input device to be pairedwith and control the selected controllable object. In anotherembodiment, the operator 1 may run a finger between the input deviceicon and the controllable object icon on a touch screen 604 to pair themtogether. In yet another embodiment, the operator 1 may use a mouse orpointer to select two icons. The display 600 may indicate theassociation (e.g., pairing) between the input device 500 and thecontrolled object for example by indicating a line between icons.

For example, one controller 514, labeled Wireless Controller 1, maycontrol one controllable object, labeled Tool 3. One controller 514,labeled Wireless Controller 2, may control one controllable object,labeled Tool 4. Other configurations between the input devices and thecontrollable objects are possible by following the selection sequencedescribed above. FIG. 37 is a screen shot of the display shown in FIG.36. For example, one controller 514, labeled Wireless Controller 1, maycontrol one controllable object, labeled Tool 1. One controller 514,labeled Wireless Controller 2, may control one controllable object,labeled Tool 5.

The control system 400 may associate the coordinate system of the inputdevice 500 and coordinate systems of the controlled objects, such as theone or more hyperdexterous surgical arms 200 and/or the one or morehyperdexterous surgical tools 300. The display 600 may show theassociations (e.g., pairings) between the input devices 500 and thecontrolled objects. The input device 500 and the controlled object maymove in the same coordinate system. The input device 500 may move in arectilinear coordinate system, which may move the controlled object alsoin a rectilinear coordinate system. For instance, a movement of an inputdevice 500 a distance in the positive x-axis direction may move ahyperdexterous surgical tool 300 a different distance or the samedistance in the positive x-axis direction. Both the controlled objectand the input device move in a rectilinear coordinate system.

The input device 500 may move about an imaginary center, which may movethe controlled object about a virtual grip 512. For instance, if thehyperdexterous surgical tool 300 moves in circular arcs about a fulcrum,moving the input device 500 in a circular motion about an imaginarycenter may be more natural. This circular motion may be more naturalthan a rectilinear motion of the input device 500. Both the controlledobject and the input device move in a polar coordinate system.

The input device 500 and the controlled object may move in differentcoordinate systems. The input device 500 may be moved in a rectilinearcoordinate system which may move the controlled object in a polarcoordinate system. For some types of motions, it may be easier tocalculate and/or display the motion of the controlled object inalternative coordinate systems. Combinations are possible.

The operator may establish certain constraints for the hyperdexteroussurgical system 100. The control system 400 can be arranged such thatthe constraints are measured quantities such as position or derivedparameters such as distance, velocity, force, and tension. The controlsystem can be arranged such that the constraints can be different foreach controlled object.

When executing the movements of the various hyperdexterous surgicaltools 300, the control system 400 may maintain one or more constraints.Constraints may be a physical or a derived parameter such as distance,velocity, force, tension, and/or radius. The control system 400 mayapply one or more constraints to the motion of one or morehyperdexterous surgical tools 300. Constraints may be different for eachcontrolled object. For example, the hyperdexterous surgical tools 300locked to a single tool may be subjected to different constrains. Eachhyperdexterous surgical tool 300 may have an independent constraint.

In some embodiments, two hyperdexterous surgical tools 300 can beconstrained to move together. The operator 1, such as the surgeon, canlock one or more of the hyperdexterous surgical tools 300 to a singletool. All of the locked hyperdexterous surgical tools 300 would followthe single tool. In other words, all of the locked hyperdexteroussurgical tools 300 would move substantially in unison (e.g. at the sametime) as the single tool is controlled by the operator 1 and moved fromplace to place. Conceptually, this lock step motion can be thought of asa virtual tether holding the relative positions of the hyperdexteroussurgical tools 300 constant and moving the set of hyperdexteroussurgical tools 300 in unison.

The concept of locking one or more hyperdexterous surgical tools 300 toa single tool applies equally to whether the single tool is ahyperdexterous surgical tool 300 or a manual tool 350. For one or morehyperdexterous surgical tools 300 to follow the manual tool 350, themanual tool 350 can be tracked in space for translation and rotation.This type of configuration where one or more hyperdexterous surgicaltools 300 follows the manual tool 350 may be useful when the operator 1,such as a surgeon, wants to move large distances within the work space.The concept of locking may also advantageously avoid collisions betweenthe hyperdexterous surgical tools 300, manual tools 350 and/or theanatomy of the patient 2.

It may not be necessary to lock all the hyperdexterous surgical tools300 to a single tool. In some embodiments, the operator 1 can choose tolock some (but not all) of the hyperdexterous surgical tools 300 to asingle tool and leave some hyperdexterous surgical tools 300 unlocked,and in place. For example the camera 304 may be in a good position butthe other hyperdexterous surgical tools 300 such as graspers may need tobe moved to a different location. The other hyperdexterous surgicaltools 300 such as graspers can be locked, so as to follow the singlehyperdexterous surgical tool 300, but the camera 304 can remain inplace. The need for moving all or some of the hyperdexterous surgicaltools 300 in unison may arise if the operator 1 needs to operate onanother section of the body distant from the current section undergoingsurgical intervention.

In some embodiments, the one or more hyperdexterous surgical tools maybe locked to a single hyperdexterous surgical tool or a single manualtool through the use of the display 600. FIG. 40 shows a screen shotthat may be displayed on the display 600. Other screen layouts arepossible. The display 600 enables the operator 1 such as a surgeon tolock one or more of the hyperdexterous surgical tools 300. The topportion of the screen shot shows the input devices 500 and thecontrollable objects, such as hyperdexterous surgical tools 300. FIG. 40shows five hyperdexterous surgical tools 300 including camera 304. Themanual tool 350 is also shown. FIG. 40 shows two input devices 500, awireless controller 520, and a wireless controller 522. Depending on thetools, including all hyperdexterous surgical tools 300 and all manualtools 350 in communication with the control system 400, the display 600displays the appropriate icons. The wireless controller 520 is connectedto the controllable object, Tool 1. The wireless controller 522 is notconnected to a controllable object.

The bottom portion of the screen shot of FIG. 40 shows locking options.Any controllable object controlled by an input device may not appear onthe list. For instance, since Tool 1 is being controlled by the wirelesscontroller 520, the display 600 does not display Tool 1 in the set oftools that can be locked. The selected controllable objects, such as oneor more hyperdexterous surgical tools 300, can be selected by the userto be locked. For instance, Tools 2 and 3 are selected to be locked,while Tools 4 and 5 are unselected. The display 600 can highlight theselected entries, as shown in FIG. 40. The hyperdexterous surgical tools300 that have been selected to be locked are shown in white whereas theothers are shown in black. On the bottom portion of the screen shot, thedisplay 600 displays the options for the single tool. The single tool isfollowed by the one or more locked hyperdexterous surgical tools 300.For instance, the manual tool 350 is selected to be the single tool,while Tool 1 controlled by the wireless controller 522 is unselected.The display 600 can highlight the selected entries, as shown in FIG. 40.In this example, Tool 2 and Tool 3 are locked to the manual tool 350,but other configurations are possible.

In some embodiments, the movement of the single tool may be controlledby smaller hand gestures rather than larger movements associated withthe set of tools. For instance, the single tool may be Tool 1,controlled by the wireless controller 522. The wireless controller 522may be worn by the operator 1, as described above. For instance, one ormore fingers of the operator's hand may have sensors affixed orotherwise attached. A specific movement of the hand or of the fingersmay move the single tool. The one or more hyperdexterous surgical tools300 that are locked will move according to the movement of the singletool, as described above.

The operator 1 such as a surgeon may engage or disengage the selectedhyperdexterous surgical tools 300 from actively following the singletool. The operator can utilize the clutch 112, such as a foot pedal. Forexample, when the clutch 112 is depressed, the selected hyperdexteroussurgical tools 300 will follow the single tool. When the clutch 112 isreleased, the selected hyperdexterous surgical tools 300 will stopmoving and remain in place. This feature may advantageously allow theoperator to reposition his or her arms much like computer usersreposition a computer mouse by lifting the mouse and placing it in amore comfortable position.

The following three constraints provide examples of additionalconstraints that may be imposed. Constant tension of the gripped objectis maintained. The ratio of motion of the hand to the motion of thehyperdexterous surgical tools is 1:10, meaning that when the hand moves10 cm, the tools move 1 cm. The camera tool 304 is rotated about afulcrum instead of translated linearly.

The control system 400 can apply a constraint to a hyperdexteroussurgical tool 300. For instance, the control system 400 can adjust theposition of one hyperdexterous surgical tool 300 in order to apply aconstant tension to the tissue held by the set of hyperdexteroussurgical tools 300. These constraints provide useful ways to control thehyperdexterous surgical tools 300. Constraints may be applied relativeto the manual tool 350 as well. Other constraints can be applied inaddition to, or in place of, those described above. The control system400 can apply a constraint between the hyperdexterous surgical tool 300and the camera 304. The control system 400 can establish position, speedor force constraints between one or more hyperdexterous surgical tool300 and the camera 304.

In FIGS. 38A-B, two hyperdexterous surgical tools 300, such as grippers,are shown on either side of a tissue outgrowth. The hyperdexteroussurgical tools 300 are at a certain distance 308 as shown in FIG. 38.When the outgrowth is removed, the hyperdexterous surgical tools 300 mayneed to be repositioned. For instance, the tissue may need to be held ata higher tension since the outgrowth is no longer pulling the tissue.The hyperdexterous surgical tools 300 are now at a distance 310, whichmay be greater than distance 308. The control system 400 may control thehyperdexterous surgical tools 300 so that certain parameters of thesurgery may be maintained. For instance, in this example, the controlsystem 400 may control the hyperdexterous surgical tools 300 so that thetissue is in constant tension.

If multiple tools are used in the hyperdexterous surgical system 100,then any set of two or more tools may be controlled as a rigid body. Forexample, the set of hyperdexterous surgical tools 300 may be controlledsuch that the tissue is manipulated as a rigid body and is tracked bythe camera 304. The rigid body effect may be achieved when the controlsystem 400 calculates and applies appropriate tension through thehyperdexterous surgical tools 300 to the tissue. The calculations may bein real time and be dynamic. This may assist in maintaining the rigidbody effect, for instance if the section of tissue is in motion due tocontrolling motions of the surgeon.

Frame of Reference, Visual Cues

The control system 400 advantageously allows the operator 1 to controlthe hyperdexterous surgical tools 300 and/or the manual tools 350naturally. There are at least two interconnected pieces that allownatural and effective control: visual cues and moving in frames ofreference that are natural to the human body and easily processed by thehuman brain. The brain can easily understand frames of referencesassociated with a person's wrist. The control system 400 advantageouslyprovides the operator 1 with visual cues related to the hyperdexteroussurgical tools 300 and/or manual tools 350, which enable the operator tobetter understand the frames of reference associated with the movementof the hyperdexterous surgical tools 300 and/or manual tools 350.

Frame of References

As discussed above, the human brain can easily understand movement ifthe frames of references are coupled to the wrist. With reference toFIGS. 41A-41D, the brain understands that in order to reach the objects,the wrists must be moved toward the object. The brain understands whichway to move the wrist based on the orientation of the wrist.

In FIGS. 41A and 41C, a left hand and a right hand of an operator 1 suchas a surgeon is shown. The direction of motion of the hands is shown oneach figure. Each hand moves toward an object. The left hand moves fromPosition 1 to Position 2, toward Object A. The right hand moves fromPosition 1 to Position 2, toward Object B. A frame of reference isplaced on each wrist. The orientation of the frame of reference isindicated by the pointed end of the icon representing the frame ofreference. The imaginary observer 802 would be standing on the circle inthe middle of the frame of reference icon. In other words, the imaginaryobserver 802 would be looking in the direction of the pointed end of theicon, in the direction of the arrow.

FIGS. 41A and 41C indicate the motion and the frame of reference foreach hand. FIGS. 41B and 41D indicate how the motion is perceived by theimaginary observer 802 placed as explained above The left hand movesfrom Position 1 to Position 2, toward Object A. The right hand movesfrom Position 1 to Position 2, toward Object B. A frame of reference isplaced on each wrist.

In FIG. 41A, both hands are oriented the same way, so both hands mustmove forward to reach the objects. In FIG. 41A, the frames of referencefor each imaginary observer 802 (one on each wrist) are oriented thesame way. In other words, the frames of references are aligned.

FIG. 41B illustrates how the objects will appear to these imaginaryobservers 802. Both imaginary observers 802 see substantially the sameimages. At Position 1, the Objects A and B, the circle and the triangle,are a distance away from the imaginary observers 802. At Position 2, theObjects A and B, the circle and the triangle, are closer to theimaginary observers 802. The imaginary observer 802 on the right wristis closer to the Object B, the triangle. The imaginary observer 802 onthe left wrist is closer to the Object A, the circle.

In FIG. 41C, the left hand is perpendicular to the right hand. The righthand must move forward to reach the objects. The left hand must movetoward the left to reach the objects. In FIG. 41C, the frames ofreference for each imaginary observer 802 (one on each wrist) are notoriented the same way. The frame of reference for the left hand isrotated 90° with respect to the frame of reference on the right hand.Although the FIG. 41C shows frames of reference that are rotated 90° toeach other, any other orientation in a three-dimensional space ispossible.

FIG. 41D illustrates how the objects will appear to these imaginaryobservers 802. Both imaginary observers 802 see a different image. Tothe imaginary observer on the right wrist, the Objects A and B are infront of the viewer. To the imaginary observer on the left wrist, theObjects A and B are to the left of the viewer. At Position 1, theObjects A and B, the circle and the triangle, are a distance away fromthe imaginary observers 802. At Position 2, the Objects A and B, thecircle and the triangle, are closer to the imaginary observers 802.

In FIGS. 41A-41D, the operator is provided with visual cues. Theoperator can see the Objects A and B. The operator 1 can see his hands.When the operator 1 is manipulating the hyperdexterous surgical tools300 and/or manual tools 350 within the patient's body, the operator 1could benefit from receiving visual cues related to the hyperdexteroussurgical tools 300 and/or manual tools 350. The visual cues can enablethe operator to better understand the frames of reference associatedwith the movement of the hyperdexterous surgical tools 300. As mentionedherein, hyperdexterous surgical system 100 enables the operator 1 tocontrol hyperdexterous surgical tools 300 and manual tools 350simultaneously. For example, as a non-limiting example, the operator 1can control a hyperdexterous surgical tool 300 such as a gripper withthe left hand and the manual tool 350 such as a stapler with the righthand. The control system 400 can allow the operator 1 to manipulate eachtool in an independent frame of reference 800. The control system 400can include a control algorithm to translate the motions of the operator(e.g., motions controlling the input device 500) into movements in thecorrect frame of reference.

Referring now to FIG. 42A, the operator 1 is manipulating thehyperdexterous surgical tool 300 and the manual tool 350 within the bodyof the patient. FIG. 42A shows the operator 1 controlling thehyperdexterous surgical tool 300 with his or her left hand and themanual tool 350 with his or her right hand. Both the hyperdexteroussurgical tools 300 and/or manual tools 350 are within the body andobstructed from the operator's view.

The manual tool 350 may dictate the position of the operator 1 relativeto the patient 2. The design of the manual tool 350 will dictate theposition of the hand controlling the manual tool 350. The operator 1 maywant to control the hyperdexterous surgical tool 300 from substantiallythe same position with his or her other hand.

The frame of reference associated with the operator's left handcontrolling the hyperdexterous surgical tool 300 may be rotated withrespect to the frame of reference associated with the operator's righthand controlling the manual tool. As shown in FIG. 42A, the frame ofreference for the right hand is rotated 90° with respect to the frame ofreference for the left hand (opposite configuration of that shown inFIG. 41A).

The frames of reference are placed on each wrist. Two orthogonal pairsof arrows are shown on each wrist to illustrate how the objects willappear to the imaginary observers 802. On the right wrist, the letters“RR” indicate that this direction is the right side of the operator'sright wrist, or what appears to be the right side to the imaginaryobserver 802. The letters “LR” indicate the right side of the operator'sleft wrist, or what appears to be the right side to the imaginaryobserver 802. Both imaginary observers 802 see a different image. Thecamera 304 (with the lens inside the body) sees both the manual tool 350and the hyperdexterous surgical tool 300 that are inside the body.

To manipulate the manual tool 350, the operator 1 may move the righthand and may use the right wrist in the frame of reference as shown. Theright hand is illustrated at a 90° angle to the right arm, and at a 90°angle to the left hand. The operator can manipulated each tool in anindependent frame of reference. The manual tool 350 can be manipulatedwith respect to a frame of reference associated with the right wrist andthe hyperdexterous surgical tool 300 can be manipulated with respect tothe left wrist.

FIG. 42A shows an input device 500 affixed to the left hand. Theoperator 1 can use the input device 500 to control a hyperdexteroussurgical tool 300 such as the grasper. The camera 304 (with the lensinside the body) will show the motion of the grasper and/or end effectorof the grasper on the display 702. In the illustrated embodiment, theoperator 1 is standing beside the patient 2 facing the patient 2 and hashis or her head facing the display 702 (e.g., facing straight aheadalong Arrow B). The control system 400 can show on the display a naturalperspective of the surgery. For instance, a rightward motion of the lefthand may be shown as a rightward motion of the hyperdexterous surgicaltool 300 as presented on the display 702. The control system 400 can usethe position of the input devices 500, the position of the camera 304,the position of the operator 1, and/or the position of thehyperdexterous surgical tool 300 to orient the image on the display 600,702. The control system 400 can calculate the motion of the end effectorof the grasper. The control system 400 can orient the image of thecamera 304 on the display 600, 702. For instance, the control system 400can orient the image such that a rightward motion of the left wrist isseen as a rightward motion of the hyperdexterous surgical tool 300.

The control of the manual tool 350 with the operator's right hand may bein a frame of reference associated with the operator's right wrist. Thecontrol of the hyperdexterous surgical tool 300 with the operator's lefthand may be in a frame of reference association with the operator's leftwrist. This frame of reference of the left wrist may be an independentframe of reference from the frame of reference of the right wrist.Regardless the angle of the operator's right wrist and right hand, thehuman brain can recognize right, left, up and down relative to the rightwrist and hand.

The two frames of reference may be completely independent, partiallyaligned, or the completely aligned. In some embodiments, thehyperdexterous surgical tool 300 may move in a frame of referencealigned with the surgical target. Both the manual tool 350 and thehyperdexterous surgical tool 300 may move with respect to the surgicaltarget to allow consistent movement of the manual tool 350 and thehyperdexterous surgical tool 300. It is to be noted that in all theseexamples, the frame of reference of the manual tool 350 may or may notbe aligned to the frame of reference of the hyperdexterous surgical tool300. Each situation will have its unique advantages in how thehyperdexterous surgical tools 300 and the manual tools 350 arecontrolled.

When controlling multiple tools in independent frames of reference,different combination of tools may be utilized. The operator may controlthe hyperdexterous surgical tool 300 and the manual tool 350, or two ormore hyperdexterous surgical tools 300, or two or more manual tools 350.The operator 1 may manipulate one or more hyperdexterous surgical tools300 with each hand, as opposed to controlling the manual tool 350 andthe hyperdexterous surgical tool 300 as described above.

The human brain is also capable of readily comprehending andcoordinating the hyperdexterous surgical tool 300 with the manual tool350, in order to use the two tools together. The brain is able tocoordinate movement despite the different frames of reference. In FIG.42A, the operator 1 is holding the manual tool 350, such as a stapler,with the right hand and the input device 500 with the left hand.

Visual Cues

The human brain is capable of determining how to move the manual tool350 and the hyperdexterous surgical tool 300 with adequate information.This information should enable the user to naturally understand themovement of the hyperdexterous surgical tool 300. The control system 400advantageously provides such visual cues to the operator 1 (e.g.,surgeon), for example, by orienting images presented to the operator orvarying the information provided in said images (e.g., illustrating atleast a portion of the body of the patient 2 to help the operator 1understand the orientation of the tools) to facilitate natural controlof the hyperdexterous surgical tools 300. In this manner, the controlsystem 400 can enable natural control of the hyperdexterous surgicaltool 300 from any location of the operator 1.

FIG. 42B shows the view of the display 702. The display 702 can provideimages captured by camera 304, shown in FIG. 42A. A rightward motion ofthe left hand along the direction of Arrow A is shown in FIG. 42A. Therelative orientation of the tools can augment the operator's 1understanding of the workspace. The display 702 can represent the manualtools 350 and the hyperdexterous surgical tools 300 in same environment.However, an image other than the camera image, such as an image of theorientation of the patient relative to the tools 300, 350, may augmentthe operator's 1 understanding of the motion of the hyperdexteroussurgical tool 300 and/or manual tool within the body of the patient.

The control system 400 may orient the camera image on the display 600,702. The displays 600,702 can show both the manual tool 350 and thehyperdexterous surgical tool 300. The control system 400 may orient thecamera image relative to the frame of reference of the operator's wrist.The control system 400 may determine the direction of the motionrelative to the operator's wrist and present the image of the movementin the same direction. For example, in FIG. 42A, the operator 1 moveshis left hand along arrow A. The operator moves his left hand toward theright.

As shown in FIG. 42B, the control system 400 can orient the image of thecamera 304 such that the image presented on the display 702 matches thedirection of the movement. When presented on the display 702, the motionof the hyperdexterous surgical tool 300 is along the original direction,rightward. From the point of view of the camera, this motion may beleftward or angled. The control system 400 can display the motion of theof the hyperdexterous surgical tool 300 such that the operator 1 is ableto understand the motion naturally. By looking at the display 702, theoperator 1 can coordinate the rightward movement of the input device 500with the rightward motion of the hyperdexterous surgical tool 300.

The orientation of the image on the display 600, 702 may assist with theuse of hyperdexterous tools or manual tools that rotate about thefulcrum. The motion of the distal end of the manual tool 350 (thesection furthest away from the hand) may move in the opposite directionrelative to the hand. In other words, if the right hand is manipulatingthe stapler about a fulcrum, then a rightward motion of the handle ofthe stapler will translate to a leftward motion of the distal end of thestapler. By providing visual cues, the hyperdexterous surgical system100 may enable natural use of hyperdexterous tools or manual tools thatrotate about the fulcrum.

The control system 400 can enable natural control of the hyperdexteroussurgical tool 300 from any location of the operator 1 via the imagespresented to the operator 1 as discussed herein. During the course ofthe procedure, the operator 1 may need to move about the operating area.The manual tool 350 may dictate the location of the operator 1. Thecontrol system 400 can present images based on the point of view of theoperator 1, regardless of the operator's location.

The position of operator 1 may be tracked. The image can be updated tobe consistent with the point of view of the operator 1. The image can becalculated based on the location of the operator 1, particularly in thezoomed out view. Tracking of the operator 1 may be accomplished by usingone of various technologies such as but not limited to affixing sensorsto the operator 1, or by using an optical localization system. Thehyperdexterous surgical system can have a global tracker that tracks theoperator 1.

The location of the operator 1 can be an input into the control system400. The control system 400 computes and presents images from the pointview of the operator 1. In order words, the display 600, 702 can showwhat anatomy or tools the operator 1 would be seeing from that location.As the operator 1 moves around, the image presented on the display 600,702 would be based on the position of the operator 1. The computationand presentation of the images based on the operator's location mayaugment the operator's understanding of the anatomy and allow theoperator to more easily interact with both manual tools 350 and thehyperdexterous surgical tools 300.

The coordinate conversion can be associated with the proximal end (e.g.,laparoscopic tools) or the distal end (e.g., end effectors). The controlsystem 400 may present images of the hyperdexterous surgical tools 300and the manual tools 350 in independent coordinate systems. As mentionedpreviously, the coordinate system of the input device 500 can bedifferent than the coordinate system of the hyperdexterous surgicaltools 300. The image on the display 600, 702 can show the motion of thehyperdexterous surgical tools 300 in the coordinate system of thehyperdexterous surgical tools 300.

FIG. 42C shows another example of an arrangement of the tools, thedisplay 702, and the operator 1. The display 702 is shown off to theleft of the operator 1. The operator 1 is standing beside the patient 2facing the patient 2 but has his or her head turned towards the display702 (along Arrow C). The frame of references of each hyperdexteroussurgical tool 300 is associated with each wrist of the operator 1. Thecontrol system 400 can enable natural control of the hyperdexteroussurgical tool 300 from any location of the display 702. The display 702can present visual cues to enable the operator to understand the motionof the hyperdexterous surgical tool 300 and the manual tools. Regardlessof the orientation of the display 702, the control system can orientthis image to augment the operator's understanding. The control system400 can show on the display 702 a natural perspective of the surgeryregardless of the positioning of the display 702. The association of theframes of reference with the wrists maintains the intuitiveness ofcontrol.

The displays 600, 702 can show the same image or different images. Thedisplay 600 can provide an input to the system, as described herein. Thedisplays 600, 702 can show multiple images on a single screen.

The display 600, 702 may show different types of association between themotion of the input device 500 and the motion of the hyperdexteroussurgical tool 300. There may be, in some embodiments, a 1:1 relationshipbetween the hand motion, the tool motion, and the motion shown on thedisplay. Other relationships are possible. The display 600, 702 may showinverse motion to the direction of the hand motion and the tool motion,such that the motion is shown in reverse. The display 600, 702 may showmotion skewed at an angle to the direction of the hand motion and thetool motion. The display 600, 702 may show any orientation with respectto the frame of reference associated with the wrist.

The wrists are mentioned only as an example of an object which the frameof a reference may be associated with. The frame of reference can beaffixed to any portion of the operator's body, the patient, objects inthe work space, objects in the operating arena, the display, thehyperdexterous surgical tools, the camera the hyperdexterous surgicalarms, or any other object. However, affixing the frame of reference tosections of the operator's body, including the forearm, wrist, hand, andhead may facilitate control of the hyperdexterous surgical tools 300 ina natural manner.

As discussed herein, the hyperdexterous surgical system 100advantageously allows the operator 1 to control the hyperdexteroussurgical tools 300 from a variety of frames of references. For example,the operator 1 can control the hyperdexterous surgical tools 300 from aframe of reference of the camera 304 in the workspace, inside the body.The operator 1 can map the movements of his hand to the movement of thehyperdexterous surgical tool in the frame of reference of the camera304. This view can be limiting for large motions or motions where it ismore natural to move with respect to a frame of reference outside thebody of the patient. Therefore, the hyperdexterous surgical system 100allows the operator to dynamically change the frame of reference toanother view. The operator 1 can switch to a world-grounded frame ofreference (e.g., a view of the operating arena and the patient) for oneor more hyperdexterous surgical tools 300. The world-grounded frame ofreference may be helpful for large motions and/or when one or morehyperdexterous surgical tools 300 are locked to a single tool, asdescribed herein. The world-grounded frame of reference may be helpfulwhen moving one or more hyperdexterous surgical tools 300 to a newlocation relative to the patient. The world-grounded frame of referencemay be helpful when the operator 1 repositions himself relative to thepatient and/or switch which hands control the input device 500 based onhis new position. The operator 1 can control the camera 304 and presentimages on the display 600, 702 of the world-grounded frame of reference.The hyperdexterous surgical system 100 allows the operator todynamically change the frame of reference to back to the frame ofreference of the camera 304.

In some embodiments, the frames of reference may be in motion. As anon-limiting example, a frame of reference may be attached to a manualtool 350 that may be in the process of being moved. The control system400 may lock one or more hyperdexterous surgical tools 300 to the manualtool 350, such that the set of tools moves together. The concept oflocking is described herein. The display 600, 702 may show a frame ofreference associated with the moving tools and may provide assurancethat the set of tools is moving as a group as intended.

A frame of reference may be established based on the position of theoperator 1 when the clutch 112 is initially engaged. When the clutch 112is engaged, the input device 500 can control one or more hyperdexteroussurgical tools 300. When the clutch 112 is disengaged, the controlsystem 400 can store this reference frame. When the clutch 112 isengaged again, the one or more hyperdexterous surgical tools 300 movewith respect to the same reference frame established earlier.

In some embodiments when the clutch 112 is engaged again, a newreference frame is established based on the new position of the operator1. The operator 1 may decide whether to use the frame of referenceestablished in the prior engagement of the clutch 112 or to use a newframe of reference. The engagement of the clutch 112 may establish oneor more frames of reference. Only one frame of reference may beestablished by the control system 400 if the operator 1 is only usingone input device 500. However, if the operator 1 is using two inputdevices 500, then two frames of references may be established by theclutch 112. The frames of reference may be associated with each wristand may be aligned, partially aligned or independent.

The right hand and the left hand of the operator 1 can manipulateobjects in two different frames of reference. The objects can bedissimilar in size, shape or function. The hyperdexterous surgicalsystem 100 allows simultaneous control of a manual tool 350 and ahyperdexterous surgical tool 300. The operator 1 is provided with enoughcues regarding the constraints on the manual tool 350 and/or thehyperdexterous surgical tool 300 to enable this simultaneous control.The operator 1 is provided with enough cues regarding the frames ofreference of the manual tool 350 and/or the hyperdexterous surgical tool300 to enable this simultaneous control. The control system 400 of thehyperdexterous surgical system 100 advantageously allows the motion ofthe hyperdexterous surgical tools 300 and/or manual tools in variousframes of reference. This ability may be very useful during the combineduse of the manual tools 350 and the hyperdexterous surgical tools 300.

Sources of Visual Cues

One source of information is visual cues as seen by the operator 1,either through observing the operating arena or the displays 600, 702.Additional information can augment the operator's understanding of themotion of the hyperdexterous surgical tool 300 and the manual tool 350.The visual cues can be supplied by the control system and shown on thedisplays 600, 702.

One source of information is visual cues as seen by the operator 1through observing the operating arena. From the operator's location, theoperator can see the set-up of the operating arena. The operator 1 cansee the orientation of his body, including his hands, relative to thepatient. The operator 1 can see the location that the tools enter thebody. In other words a surgeon may use objects around him or her such asthe bed 102, the patient 2, his or her hands as cues to understand theposition of the tools 300, 350 in relation to the anatomy. The surgeoncan manipulate the hyperdexterous surgical tool 300 or the manual tool350 consistent with that understanding.

One source of information is visual cues is images presented on thedisplay 600, 702. The control system 400 can compute and present imagesrelevant to the operator's understanding of the procedure. The imagescan enable the operator 1 to see the hyperdexterous surgical tools 300,the manual tools 350, and/or the patient's anatomy. The images canoriginate from one or more visualization components of thehyperdexterous surgical system 100. These components include one or morecameras 304, which can be controlled by the control system 400. Thecamera 304 can be considered a hyperdexterous surgical tool 300 andmoved by a hyperdexterous robotic arm 200 to provide images to theoperator 1. For instance, multiple cameras 304 may be deployed. Eachcamera 304 may acquire images of a different section of the anatomy. Insome embodiments, millimeter-sized cameras 304 are placed onto eachtrocar tip. In some embodiments, the control system 400 and/or thevisualization components perform real-time 3D reconstruction of theenvironment both internal and external to the patient 2. The controlsystem 400 and/or the visualization components can integrate priorimaging of the patients, such as prior x-rays and CT scans. Informationfrom sources can be blended to give more complete information to theoperator 1. The visualization system 700 may provide the operator 1 thefreedom to view the work space from various sources.

The images can be viewed on the one or more displays 600, 702. Thedisplay 600 may be configured to receive feedback from the operator 1,as described herein. The images can be viewed on the one or moreimmersive consoles 704. The hyperdexterous surgical system 100 allowsthe operator 1 to move around and place him or herself in the mostoptimal position in relation to the patient 2 for the procedure. Duringthe procedure, the operator 1 (e.g., surgeon) may reposition himself orherself. The one or more displays 600, 702 allow the operator 1 to viewthe hyperdexterous surgical tools 300, the manual tools 350, and/or thepatient's anatomy from multiple locations. The image on the display 600,702 may be updated based on the location of the operator 1.

The image shown on the display 600, 702 may be dependent on the type ofmanipulations being performed. For example, if the operator 1 is doingdelicate suturing, a zoomed in view of the anatomy and the tools may beshown on the display 600, 702. This view may be obtained directly fromthe camera 304. If the operator now wants to move to a different part ofthe body, gross motions of the tools are required, a zoomed out view maybe shown on the display 600, 702. In other words, the zooming factor canadapt to the motion. The zooming function may be accomplishedautomatically by the control system 400 based at least in part on thetype of motion being performed. The zooming function may also beinitiated by the operator 1. The zoom levels may be changed using theinput devices 500, such as by using gestures to zoom in or zoom out.Other ways of changing zoom levels are possible, such as attachingmanual devices such as thumbwheels on the input device 500 where theoperator can move the thumbwheel to change the zoom level. Other waysinclude providing buttons or slide bars on the display 702 or thedisplay 600.

During zooming operations, the transition between images can be smoothand seamless. As the images zoom out, fewer anatomical details may bedisplayed. The control system may change the image feed, such as changefrom the camera 304 within the patient 1 to the feed from camera 304mounted outside the patient's body. Other sources of data for the zoomedout view are discussed below.

In some embodiments, a virtual camera 706 (see FIG. 43) may be createdthus enabling target visualization from various points of view. Thevirtual camera 706 creates an image from any point of view. The virtualcamera 706 may be associated with the point of view of any of themultiple controllable objects present around the work space. The virtualcamera 706 may be associated with the point of view of the operator 1such as the surgeon. As the operator 1 moves around, the visualizationsystem 700 adjusts to the current point of view of the operator. Thevirtual camera 706 can create an image using multiple sources, asdescribed herein.

In some embodiments, the virtual camera 706 may be associated with thecamera tool 304. The operator 1 such as a surgeon may choose to adjustthe position of the camera 304 during surgery, therefore adjusting thevirtual camera 706. Therefore, the display 600, 702 will be updated asthe camera 304 moves.

FIG. 43 shows an example of how a virtual camera 706 may be adjusted bythe operator 1 such as a surgeon during surgery. The operator 1 can betracked if he or she is wearing a tracking device. If the view of thecamera 304 is inverted to that of the operator 1, the control system 400can recognize the position of the operator 1 and invert the virtualcamera 706 on the display 600, 702 to better reflect the point of viewof the operator 1. As the operator 1 moves to be aligned with the camera304, the control system 400 can recognize the position of the operator 1and reflect the true image of the camera 304. Images may be inverted,rotated, and left-right flipped on the display 600, 702 to reflect theviewpoint of the tracked objects, such as the operator 1, orcontrollable objects such as hyperdexterous surgical arms 200 and/orhyperdexterous surgical tools 300. As another example, the display 600may display sliders or buttons to change the position of the virtualcamera 706 so that the operator 1 such as a surgeon may choose the mostappropriate view.

Various camera parameters may be controlled and adjusted to enhance theimage from the virtual camera 706. As a non-limiting example, the zoomfunction may be adjusted. For example, if large scale motions aredesired, the image on the display 600, 702 may be zoomed out. As theoperator 1 such as a surgeon maneuvers the tools for the large scalemotion, the image on the display 600, 702 can zoom in and out to viewthe work space. This can be done automatically. The camera parameterssuch as angle and zoom may be adjusted using hand motions, for instanceif the virtual camera 706 is controlled by an input device 500. Theinput device 500, such as sensors attached to the hand or wirelesscontrollers, allow the advanced visualization system 700 to recognize apattern of motion and perform functions such as change virtual angle andzoom.

The hyperdexterous surgical system 100 may have multiple displays 600,702. Depending on the zoom levels, the control system 400 may displaydifferent images on each display 600, 702, each with differentparameters such as different camera angle. This may allow an operator 1,such as a surgeon and/or a surgical assistant, to operate simultaneouslyon the patient and each refer to the display 600, 702 which presents themost natural point of view of the surgical work space relative to eachperson's location. This could be useful for example if an assistant islocated close to the patient's legs and the primary surgeon is locatedat the patient's side. The assistant's display 702 would show theanatomy and tools from the point of view of the assistant, and thesurgeon's display 702 would show the anatomy and tools from the point ofview of the surgeon.

The parameters such as camera angle and the perspective of the imagesshown in each display 600, 702 may be dependent on one or moreparameters including the position and orientation of the patient 2, theposition and orientation of the display 600, 702, the position andorientation of the observer of the images such as the operator 1. Thisimplies that the control system 400 has knowledge of the location andorientation of the various objects, such as the display 600, 702, theoperator 1, and the patient 2. If such knowledge is not available, suchas not knowing where the operator 1 is located, the control system 400will calculate the images without that parameter.

As the operator 1 standing by the bedside looks at the patient 2 andsubsequently looks up or towards the display 600, 702, he or she mayfind it useful to navigate the anatomy or tools in a zoomed out view.The zoomed out view may be created with data from various sources. Thesesources may include live data from cameras 304 attached in one ormultiple locations around hyperdexterous surgical system 100 and/orattached to one or more hyperdexterous surgical tool 300. These sourcesmay include pre-operative imaging data such as MRI or CT data or modelsof the anatomy. For example, the process of zooming out may start withdisplaying the images of the detailed internal anatomy as seen by acamera 304, which may be inside the patient's body placed through aport. As the zoom factor decreases (i.e. the camera is zoom out), theentire organ is displayed. As the zoom factor is decreased further, thepoint of view may move outside the body and may show the outside of thepatient's body mixed in with a rendering of the internal organs. Thisconcept is illustrated in FIG. 44.

Starting with FIG. 44A, a zoomed in view of a section of the esophagusand the entry into the stomach (the gastro-esophageal opening) isillustrated. This is a typical site for bariatric surgery where thestomach is bypassed. In FIG. 44B, the outline of the stomach and someparts of the esophagus is shown. The image shown in FIG. 44B is zoomedout with respect to FIG. 44A. In FIG. 44C, the stomach, esophagus isshown in relation to the whole body. The level of zoom can be adjustedto enable the surgeon to understand how best to manipulate thehyperdexterous surgical tools 300 and/or the manual tools 350. If theoperator 1 only needs to visualize the end effectors, for instance forprecise and small motion, then a “zoomed in” image such as in FIG. 44Amay be useful. If the operator 1 wants to reposition the tools and use adifferent angle of approach to the anatomy, then an image such as FIG.44C may be useful during the process of repositioning.

The hyperdexterous surgical system 100 enables a user to move from azoomed in view inside the patient to a zoomed out view outside of thepatient. The surgeon can move to new position when the image is in thezoomed out view. The operator 1 may find it easier to reposition himselfrelative to the patient in the zoomed out view. From this position, theoperator 1 can zoom in to see the tools inside the body, as viewed fromhis new position. In some embodiments, the operator 1 can disengage theinput device 500 before he repositions himself. The operator 1 canengage the input device 500 after he repositions himself. In someembodiments, the operator 1 can switch which hands control the inputdevice 500 based on the new position.

The rendering of the internal organs may be a combination of varioussources of data. These sources include actual, real-time images as seenby cameras or other visualization devices. These sources can includethree-dimensional model data of the organs generated from thestereoscopic laparoscopic camera feed over the course of the surgery.These sources can include pre-operative data from MRI, CT or otherimaging modality. The model may be corrected and enhanced as new datafrom a real-time source, such as the camera 304, becomes available. Thecorrected model could be then displayed.

The control system 400 may categorize the data into various classes suchas, but not limited to, real time data (from the cameras 304), modeldata, pre-op data, stale data (specifically data from camera 304 thatwas taken prior to the current moment in time). Each type of data may bedisplayed differently in the blended image. For example, the stale datamay be blended in with a shade of yellow indicating caution must be usedin using that specific part of the data as it appears in the image. Inanother example, the model data may be blended in with a shade of redindicating extreme caution must be used in using that specific data asit appears in the image. This type of categorization and display mayserve as warnings and reminders to the surgeon as he or she maneuversthe tools inside a patient's body while looking at images on the display600, 702.

As the zoom factor and/or the point of view are adjusted, the controlsystem 400 may change the relationship between the motion of theoperator 1 (the controlling motion) and the motion of the hyperdexteroussurgical tools 300 (the controlled motion). As an example, in the zoomedin view, the control system 400 may scale the motion of the hands insuch a way that only small and precise motions are possible with largemotions of the hand. As the images are zoomed out, the scaling betweenthe controlling motion and the controlled motion may change, forinstance such that, in some embodiments, there is a 1:1 relationshipbetween the two. Other aspects may change according to the zoom factorsuch as the location of the virtual grip and the control points.

FIG. 45 shows an image as view on the display 702. The display 600and/or display 702 can display features of the operating arena. Thedisplay 600 and/or display 702 can depict an image of one or morehyperdexterous surgical arms 200, one or more hyperdexterous surgicaltool 300, one or more manual tool 350, the patient 2, the fixture (e.g.,the bed), the operator 1, etc. The display 600 and/or display 702 candepict the control points of the hyperdexterous surgical system 100

The display 600 and/or display 702 may show the constraints of thehyperdexterous surgical system 100. For instance, the display 600, 702may show the location of a fulcrum of a manual tool 350. The display600, 702 can present the constraint to augment the operator'sunderstanding. For instance, the display 600, 702 may show the locationof a virtual grip 512 of a hyperdexterous surgical tool 300. The display600, 702 can present the constraint to augment the operator'sunderstanding.

The display 600 and/or display 702 may show the control points of thehyperdexterous surgical system 100. The control points 2600 arelocations on the hyperdexterous surgical arm 200 and the hyperdexteroussurgical tool 200 which have the ability to move. The control system 400can cause movement about control points 2600 based upon an input of theoperator 1 or a constraint. For instance, the input by the operator 1effects the movement of the one or more sections that are connected tothe control point 2600. Moving the control point 2600 causes the one ormore sections of the hyperdexterous surgical arm 200 connected to thecontrol point 2600 to move.

The control points 2600 can be moved via the input device 500. Themovement of the input device 500 would cause movement of the selectedcontrol point 2600. This would cause movement of the hyperdexteroussurgical tool 300 that is controlled by the hyperdexterous surgical arm200. The display 600 may be used to assign an input device 500 to aspecific control point 2600.

Control points 2600 may be indicated in various ways including but notlimited to colors, cross hairs, other icons in the display. The image ofthe hyperdexterous surgical arm 200 on the display 600, 702 may be anactual camera image from one or many cameras 302 around the surgicalsite. The image of the hyperdexterous surgical arm 200 on the display600, 702 may be a drawing.

An Embodiment of a Surgical Method

In some embodiments, one or more manual tools 350 are used inconjunction with one or more hyperdexterous surgical tools 300 in thesame work space. An example of a manual tool 350 is a stapler 354, seeFIG. 39. The stapler 354 may be used in conjunction with thehyperdexterous surgical tool 300. The workflow when using both types oftools in a colon resection surgical procedure is shown in FIG. 39, whichshows a method 5. FIG. 39 illustrates one method of using ahyperdexterous surgical tool 300 and a manual tool 350. The methodrelates to holding the colon in a particular position and placing astaple line across the colon. The system includes a first grasper 312, asecond grasper 314, a camera 304, and a stapler 354. The system includestwo input devices 500, a first controller 516 and a second controller518.

In step 10, the operator 1 may position the camera 304 by using any ofthe input devices 500. In some embodiments, the operator 1 connects anicon of the input device 500 with an icon of the controllable objectsuch as the camera 304.

Then, in step 20, the operator 1 uses the first controller 516, which iscontrolled by one hand of the operator, to move a controllable object,such as the first grasper 312. The operator uses the first grasper 312to position and hold a section of the colon. The operator uses theclutch 112 to disengage the controllable object, the first grasper 312.The first grasper 312 will remain in place. The operator 1 can set firstcontroller 516 down.

Further, in step 30, the operator 1 can pick up the manual stapler 354with one hand. The operator 1 can position the manual stapler 354 andmove the stapler 354 to the targeted position, as seen by the positionedcamera 304. In step 40, the operator 1 uses the second controller 518,which is controlled by the hand not holding the stapler 354. The secondcontroller 518 moves a second controllable object, such as the secondgrasper 314. The operator 1 uses the second grasper 314 to position andhold a section of the colon. Then, in step 50, the operator 1manipulates the stapler 354 and the second grasper 314 to position thecolon in the most optimal position to receive the staples. Finally, instep 60, the operator 1 operates the manual stapler 354 and the staplesare delivered to the targeted location. This method illustrates how oneor more manual tools 350 and one or more the hyperdexterous surgicaltools 300 may be used at the same time by the same operator 1 in thesame work space. The operator 1 may be standing by the patient at alltimes (e.g., at one or more, for example a plurality of, bedsidelocations) during this procedure. In some embodiments, the operator 1could perform the first two steps, steps 10 and 20, remotely, such asfrom the controller 514 remote from the patient.

With hyperdexterous surgical tools 300, it is often difficult to conveythe feeling of touch. Surgeons sometimes use touch to get betterinformation about the anatomy. However, many surgeons prefer to touchthe patent in certain situations. In some embodiments, the manual tool350 may be used to make contact with the tissue. This is done commonlywith traditional surgery. The sensory input provided by using the manualtool 350 may guide the surgeon's manipulations of the hyperdexteroussurgical tools 300. Proxy devices such as force sensors and load cellscan convey a feeling of pressure or touch through complex mechanisms tothe hyperdexterous surgical tools 300. The pressure conveyed by thehyperdexterous surgical tools 300 to the operator (via the one or moreinput devices 500, such as by communicating a signal from a transmitterof the control system 400 to a receiver of the input device 500)facilitates the understanding of the anatomy of the patient 2.

The manual tools 350 may be used in other methods, in conjunction withthe hyperdexterous surgical tools 300. For instance, the hyperdexteroussurgical arm 200 may hold the trocar 302, as shown in FIG. 2. The manualtool 350 or the hyperdexterous surgical tool 300 can be inserted throughthe trocar 302. This method of use may be beneficial for example whentissue needs to be held in place by the manual tool 350 while thesurgeon manipulates the other tools. In another example, in obesepatients the manipulation of tools becomes difficult due to the bodyhabitus; in these cases the manual tool or the hyperdexterous surgicaltool 300 can be held or supported by the hyperdexterous surgical arms200 to relieve the physical stress on the operator 1.

In embodiments herein, the hyperdexterous surgical system is describedas being coupled to a fixture. The hyperdexterous surgical system can becoupled to a bed, hospital bed, operating table, examination table,platform, floor, wall, cart, or dolly. Where the fixture is a cart ordolly, the fixture can be anchored to the floor. The fixture can belocated within a medical office, a medical examiner's office, ahospital, a doctor's office, a clinical office, or any other locationsuitable for use of the hyperdexterous surgical system.

Although described in certain embodiments in connection with surgicalprocedures, the hyperdexterous surgical system can be used in anyappropriate manner. The hyperdexterous surgical system can be used in amethod that manipulates hyperdexterous surgical tools for percutaneousinsertion. The hyperdexterous surgical system described herein can beoperated in non-percutaneous procedures (e.g., procedures that do notinvolve making incisions and inserting the hyperdexterous surgical toolspercutaneously). For example, the hyperdexterous surgical system can beused to take skin biopsies. The hyperdexterous surgical system can beused for any surgery. The hyperdexterous surgical system can be used inany appropriate medical procedure. The hyperdexterous surgical systemcan be used in conjunction with living patients (e.g., surgery) orcadavers (e.g., autopsies). The embodiments described herein can be usedin any appropriate manner. The hyperdexterous surgical arm can be usedin manufacturing or assembly of products (e.g., on an assembly line, ina clean room, etc.).

Although this disclosure has been described in the context of certainembodiments and examples, it will be understood by those skilled in theart that the disclosure extends beyond the specifically disclosedembodiments to other alternative embodiments and/or uses and obviousmodifications and equivalents thereof. In addition, while severalvariations of the embodiments of the disclosure have been shown anddescribed in detail, other modifications, which are within the scope ofthis disclosure, will be readily apparent to those of skill in the art.It is also contemplated that various combinations or sub-combinations ofthe specific features and aspects of the embodiments may be made andstill fall within the scope of the disclosure. For example, featuresdescribed above in connection with one embodiment can be used with adifferent embodiment described herein and the combination still fallwithin the scope of the disclosure. It should be understood that variousfeatures and aspects of the disclosed embodiments can be combined with,or substituted for, one another in order to form varying modes of theembodiments of the disclosure. Thus, it is intended that the scope ofthe disclosure herein should not be limited by the particularembodiments described above. Accordingly, unless otherwise stated, orunless clearly incompatible, each embodiment of this invention maycomprise, additional to its essential features described herein, one ormore features as described herein from each other embodiment of theinvention disclosed herein.

Features, materials, characteristics, or groups described in conjunctionwith a particular aspect, embodiment, or example are to be understood tobe applicable to any other aspect, embodiment or example described inthis section or elsewhere in this specification unless incompatibletherewith. All of the features disclosed in this specification(including any accompanying claims, abstract and drawings), and/or allof the steps of any method or process so disclosed, may be combined inany combination, except combinations where at least some of suchfeatures and/or steps are mutually exclusive. The protection is notrestricted to the details of any foregoing embodiments. The protectionextends to any novel one, or any novel combination, of the featuresdisclosed in this specification (including any accompanying claims,abstract and drawings), or to any novel one, or any novel combination,of the steps of any method or process so disclosed.

Furthermore, certain features that are described in this disclosure inthe context of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresthat are described in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations, one or more features from a claimedcombination can, in some cases, be excised from the combination, and thecombination may be claimed as a subcombination or variation of asubcombination.

Moreover, while operations may be depicted in the drawings or describedin the specification in a particular order, such operations need not beperformed in the particular order shown or in sequential order, or thatall operations be performed, to achieve desirable results. Otheroperations that are not depicted or described can be incorporated in theexample methods and processes. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the described operations. Further, the operations may berearranged or reordered in other implementations. Those skilled in theart will appreciate that in some embodiments, the actual steps taken inthe processes illustrated and/or disclosed may differ from those shownin the figures. Depending on the embodiment, certain of the stepsdescribed above may be removed, others may be added. Furthermore, thefeatures and attributes of the specific embodiments disclosed above maybe combined in different ways to form additional embodiments, all ofwhich fall within the scope of the present disclosure. Also, theseparation of various system components in the implementations describedabove should not be understood as requiring such separation in allimplementations, and it should be understood that the describedcomponents and systems can generally be integrated together in a singleproduct or packaged into multiple products.

For purposes of this disclosure, certain aspects, advantages, and novelfeatures are described herein. Not necessarily all such advantages maybe achieved in accordance with any particular embodiment. Thus, forexample, those skilled in the art will recognize that the disclosure maybe embodied or carried out in a manner that achieves one advantage or agroup of advantages as taught herein without necessarily achieving otheradvantages as may be taught or suggested herein.

Conditional language, such as “can,” “could,” “might,” or “may,” unlessspecifically stated otherwise, or otherwise understood within thecontext as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements, and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements, and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements, and/or steps areincluded or are to be performed in any particular embodiment.

Conjunctive language such as the phrase “at least one of X, Y, and Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to convey that an item, term, etc. may beeither X, Y, or Z. Thus, such conjunctive language is not generallyintended to imply that certain embodiments require the presence of atleast one of X, at least one of Y, and at least one of Z.

Language of degree used herein, such as the terms “approximately,”“about,” “generally,” and “substantially” as used herein represent avalue, amount, or characteristic close to the stated value, amount, orcharacteristic that still performs a desired function or achieves adesired result. For example, the terms “approximately”, “about”,“generally,” and “substantially” may refer to an amount that is withinless than 10% of, within less than 5% of, within less than 1% of, withinless than 0.1% of, and within less than 0.01% of the stated amount. Asanother example, in certain embodiments, the terms “generally parallel”and “substantially parallel” refer to a value, amount, or characteristicthat departs from exactly parallel by less than or equal to 15 degrees,10 degrees, 5 degrees, 3 degrees, 1 degree, 0.1 degree, or otherwise.

The scope of the present disclosure is not intended to be limited by thespecific disclosures of preferred embodiments in this section orelsewhere in this specification, and may be defined by claims aspresented in this section or elsewhere in this specification or aspresented in the future. The language of the claims is to be interpretedbroadly based on the language employed in the claims and not limited tothe examples described in the present specification or during theprosecution of the application, which examples are to be construed asnon-exclusive.

1. (canceled)
 2. A surgical system, comprising: at least oneelectromechanical arm selectively coupleable to a fixture, the at leastone electromechanical arm comprising a redundant degree of freedomcomprising a combination of one or more electromechanical jointsconfigured to permit motion of the at least one electromechanical armwhile preserving a remote center substantially fixed in space; at leastone electromechanical tool supported by the at least oneelectromechanical arm to define an electromechanical arm and toolassembly; a control system configured to electronically communicate withand control the operation of the electromechanical arm and toolassembly; and at least one portable handheld controller actuatable tocommunicate a control signal to the electromechanical arm and toolassembly, wherein the electromechanical arm and tool assembly isconfigured to couple to a bed via the fixture such that theelectromechanical arm and tool assembly is selectively positionable inany of a plurality of vertical positions relative to the bed and any ofa plurality of laterally outward positions relative to the bed, whereinthe redundant degree of freedom and the fixture permit repositioning ofthe at least one electromechanical arm relative to a patient on the bed,to increase free space around the patient and thereby enable a user tostand adjacent the bed and simultaneously control the electromechanicaltool via the at least one portable handheld controller and a manualtool.
 3. The surgical system of claim 2, wherein the at least oneelectromechanical arm has three degrees of freedom and comprises anelectromechanical joint comprising a redundant roll mechanism proximatea base of the at least one electromechanical arm, thus allowing theelectromechanical arm and tool assembly to be moved away from a patientduring a surgical procedure to thereby increase the workspace outsidethe patient.
 4. The surgical system of claim 2, wherein theelectromechanical arm and tool assembly is configured to achieve adesired position for the at least one electromechanical tool via twodifferent poses of the at least one electromechanical arm.
 5. Thesurgical system of claim 2, wherein the at least one electromechanicalarm is configured to maintain a tool tip position of the at least oneelectromechanical tool and the remote center while reconfiguring thecomponents of the at least one electromechanical arm in different poses.6. The surgical system of claim 2, wherein the electromechanical jointscomprise a roll mechanism, a pitch mechanism, and a redundant rollmechanism.
 7. The surgical system of claim 6, wherein the redundant rollmechanism is positioned below the remote center.
 8. The surgical systemof claim 2, wherein the control system is configured to communicate oneor more control signals to the one or more electromechanical joints toreconfigure the components of the at least one electromechanical arm indifferent poses.
 9. The surgical system of claim 2, wherein theredundant degree of freedom comprising the combination of one or moreelectromechanical joints is configured to be controlled by the controlsystem.
 10. The surgical system of claim 2, wherein the redundant degreeof freedom comprising the combination of one or more electromechanicaljoints is configured to be controlled by a user interface device. 11.The surgical system of claim 2, further comprising a support armcoupling the at least one electromechanical arm to the fixture, thesupport arm comprising a plurality of links rotatable in a combinationof planes.
 12. The surgical system of claim 11, wherein the support armcomprises one or more motors to move the plurality of links.
 13. Thesurgical system of claim 7, wherein the remote center is theintersection of a roll axis of the roll mechanism, a pitch axis of thepitch mechanism, and a tool shaft axis of the at least oneelectromechanical tool.
 14. A surgical system, comprising: at least oneelectromechanical arm selectively coupleable to a fixture, the at leastone electromechanical arm comprising a redundant degree of freedomcomprising a combination of one or more electromechanical joints; atleast one electromechanical tool supported by the at least oneelectromechanical arm to define an electromechanical arm and toolassembly; a control system configured to electronically communicate withand control the operation of the electromechanical arm and toolassembly; and at least one portable user interface device actuatable tocommunicate a control signal to the electromechanical arm and toolassembly, wherein the electromechanical arm and tool assembly isconfigured to couple to a bed via the fixture such that a base of the atleast one electromechanical arm is selectively positionable in any of aplurality of vertical positions relative to the bed and any of aplurality of laterally outward positions relative to the bed, whereinthe redundant degree of freedom and the fixture permit repositioning ofthe at least one electromechanical arm relative to a patient on the bed,to increase free space around the patient and thereby enable a user tostand adjacent the bed and simultaneously control the electromechanicaltool via the at least one portable user interface device and the manualtool.
 15. The surgical system of claim 14, wherein the at least oneelectromechanical arm has three degrees of freedom and comprises anelectromechanical joint comprising a redundant roll mechanism proximatethe base of the at least one electromechanical arm, thus allowing theelectromechanical arm and tool assembly to be moved away from a patientduring a surgical procedure to thereby increase the workspace outsidethe patient.
 16. The surgical system of claim 14, wherein theelectromechanical arm and tool is configured to achieve a desiredposition for the at least one electromechanical tool via two differentposes of the at least one electromechanical arm.
 17. The surgical systemof claim 14, wherein the at least one electromechanical arm isconfigured to maintain a tool tip position of the at least oneelectromechanical tool and a remote center while reconfiguring thecomponents of the at least one electromechanical arm in different poses.18. The surgical system of claim 14, wherein the electromechanicaljoints comprise a roll mechanism, a pitch mechanism, and a redundantroll mechanism.
 19. The surgical system of claim 18, wherein theredundant roll mechanism is positioned below a remote center.
 20. Thesurgical system of claim 14, wherein the control system is configured tocommunicate one or more control signals to the one or moreelectromechanical joints to reconfigure the components of the at leastone electromechanical arm in different poses.
 21. The surgical system ofclaim 14, wherein the redundant degree of freedom comprising thecombination of one or more electromechanical joints is configured to becontrolled by the control system.